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PREFACE 


As  introduced  by  Xerox,  benchmarking  is  a  process  for  finding  the  world-class  standards 
for  a  product,  service,  or  system  and  then  adjusting  one's  own  products,  services,  and 
systems  to  exceed  those  standards.  These  world-class  standards  can  be  found  by  looking 
at  competitors  who  are  recognized  leaders  for  the  product,  service,  or  system. 
Benchmarking  done  using  competitors  as  models  is  called  competitive  or  strategic 
benchmarking.  Operating  units  or  functions  within  a  company  can  also  be  used  as  the 
model;  this  approach  is  called  internal  benchmarking.  Finally,  benchmarking  can  be  done 
using  companies  that  are  the  best  practitioners  of  a  particular  function,  regardless  of  what 
industry  the  exemplar  is  in.  This  type  of  benchmarking  is  called  functional  benchmarking. 
Benchmarking  is  a  method  now  commonly  used  by  companies  as  part  of  their  efforts  to 
improve  quality,  and  it  is  a  necessary  element  for  winning  the  Malcolm  Baldrige  National 
Quality  Award. 

Another  form  of  benchmarking  is  called  normative  benchmarking.  In  normative 
benchmarking,  a  consultant  collects  data  from  a  group  of  companies  on  a  product,  service, 
or  process  and  then  delivers  a  set  of  statistics  to  the  companies  with  company  names 
expunged.  The  benchmarking  effort  described  herein  incorporated  elements  of  normative 
and  functional  benchmarking. 

The  Reliability  Analysis  Center  (RAC)  conducted  a  fact-finding,  study  project  to 
benchmark  the  reliability  and  maintainability  (R&M)  practices  used  by  commercial 
industry.  The  project  was  performed  for  the  Office  of  the  Under  Secretary  of  Defense 
for  Economic  Security,  Weapon  Support  Improvement  Group.  Initially,  the  study 
objectives  were  to: 

•  Gain  an  understanding  of  the  best  R&M  practices  being  used  by  commercial 
industry 

•  Recommend  changes  to  existing  DoD  R&M  standards  and  handbooks 

•  Identify  R&M  tools  and  techniques  and,  when  possible,  evaluate  their 
effectiveness,  applications,  and  cost. 

As  Defense  Acquisition  Reform  progressed,  the  second  and  third  objectives  were 
deferred.  The  project  was  rescoped  to  focus  on  the  first  objective  and  on  reliability  as  a 
means  of  providing  useful  input  to  the  ongoing  DoD  acquisition  reform  effort. 

The  benchmarking  consisted  of  four  distinct  tasks:  a  literature  search,  a  survey  of  the 
R&M  practices  of  a  wide  range  of  commercial  companies,  personal  interviews  of  a  smaller 
group  of  companies,  and  an  analysis  of  the  data  collected.  Based  on  the  results  of  these 
tasks,  areas  of  commonality  and  divergence  among  commercial  reliability  practices  were 
identified,  as  well  as  the  general  commercial  approach  to  designing,  developing,  and 
manufacturing  reliable  products.  Three  benchmarks  and  eight  Keys  to  Success  were 
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derived  from  the  findings  and  conclusions.  Insights  were  gained  into  the  motivations 
behind  commercial  companies’  approaches  and  their  use  of  military  specifications, 
standards,  and  handbooks, 


This  document  is  organized  into  five  chapters  and  six  appendixes: 


Chapter  1  -  Introduction  -  The  background  leading  up  to  the 

benchmarking  effort  is  described  as  is  the 
benchmarking  process. 

Chapter  2  -  Technical  Approach  -  The  approach  used  to 

perform  the  benchmarking  is  described. 


Chapter  3 


Results  -  The  results  of  each  step  in  the 
benchmarking  effort  are  presented. 


Chapter  4 
Chapter  5 


Appendix  A 


Conclusions  -  Conclusions  are  presented. 

Areas  Requiring  Additional  Research  -  Areas 
related  to  reliability  engineering  and  its 
implementation  that  require  further  research  are 
presented. 

Defense  Acquisition  Reform  -  Background 
information  on  the  current  reform  efforts. 


Appendix  B 


Literature  Search  Results  -  Bibliography  and 
synopses  of  selected  references. 


Appendix  C 
Appendix  D 


Appendix  E 


Survey  Forms  -  Samples  of  the  survey  forms  used. 

Interview  Questions  and  Notes  -  Samples  of  the 
interview  questions  and  expurgated  notes  from  the 
interviews. 
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1.0  Introduction 


The  document  describes  the  results  of  a  study  conducted  by  the  Reliability  Analysis 
Center  (RAC)  for  the  Office  of  the  Assistant  Secretary  of  Defense  (OASD)  for  Economic 
Security  (ES),  Weapon  Support  Improvement  Group  (WSIG).  Initially,  the  study 
objectives  were  to: 

•  Gain  an  understanding  of  the  best  R&M  practices  being  used  by  the 
commercial  sector 

•  Recommend  changes  to  existing  DoD  R&M  standards  and  handbooks  based 
on  the  best  commercial  practices 

•  Identify  R&M  tools  and  techniques  and,  when  possible,  evaluate  their 
effectiveness,  applications,  and  cost 

To  provide  timely  input  to  efforts  within  DoD  to  examine  current  acquisition  policies  and 
streamline  the  acquisition  process,  the  project  was  rescoped  to  focus  on  gaining  an 
understanding  of  the  reliability  practices  being  used  by  commercial  industry.  So,  although 
R&M  is  sometimes  used  in  this  report,  the  focus  of  the  effort  was  on  reliability. 

The  report  does  not  address  related  issues,  such  as  whether  government  specifications 
and  standards  should  be  used  at  all  or  how  to  streamline  the  government  acquisition 
process.  These  issues  are  being  investigated  by  other  government-industry  groups.  It  is, 
however,  important  to  view  the  report  in  the  context  of  Defense  Acquisition  Reform. 
Consequently,  a  discussion  of  Defense  Acquisition  Reform  is  provided  in  Appendix  A. 

1.1  BACKGROUND 

Prior  to  the  Second  World  War,  weapons  were  relatively  simple  in  capability  and 
complexity.  Most  portions  of  a  system  seldom  failed  and  when  they  did  were  easily 
fixed.  1  Due  to  technological  advances  made  during  the  war,  military  weapons  evolved  into 
extremely  complex  systems.  Complexity  was  a  byproduct  of  efforts  to  build  more 
capable,  effective  systems  to  counter  more  lethal  threats;  efforts  that  were  successful  as 
evidenced  by  systems  such  as  airborne  radar  and  the  P-61  all-weather  fighter. 
Complexity  created  new  problems  for  the  operators  who  discovered  that  they  not  only 
required  more  training  to  use  the  systems,  but  that  the  systems  were,  to  some  extent, 
more  fragile.  That  is,  the  systems  tended  to  fail  more  frequently  and  required  more 
maintenance  (more  in  terms  of  frequency  and  required  skills) .  From  this  new  reality,  the 
Department  of  Defense  (DoD)  created^  the  disciplines  of  reliability  and  maintainability 
(R&M)  engineering.  With  the  assistance  of  forums,  such  as  the  Advisory  Group  on 

^From  our  perspective.  At  the  time,  systems  such  as  the  P-40  were  considered  very  complex  and  maintenanee  erews  surely  did  not 
consider  their  job  easy.  With  apologies  to  our  predecessors,  the  fact  remains  that  technology  changed  more  during  WW  11  than  it 
had  during  the  previous  50  years,  and  the  rate  of  change  has  since  accelerated. 

•^Or  at  least  formalized  them.  During  WW  II,  the  V-1  missile  project  team,  led  by  Dr.  Von  Braun,  developed  what  was  probably 
the  first  reliability  prediction  model.  The  team  consulted  with  Eric  Pieruschka  who  asserted  that  if  the  probability  of  survival  of  an 
element  is  1/x  then  the  probability  that  a  set  of  n  identical  elements  will  survive  is  (1/x)’^.  The  formula  derived  from  this  assertion  is 
sometimes  called  Lusser's  law  (Robert  Lusser  is  considered  a  pioneer  of  reliability)  but  is  more  frequently  known  as  the  reliability 
of  a  series  system:  Rg  ~  Rj  x  R2  x  ...  x  R^. 
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Introduction 


Reliability  of  Electronic  Equipment,  the  DoD  developed  standards  and  other  guidance 
documents  on  R&M,  beginning  in  the  1950s.  Since  then,  the  specific  tasks,  methods,  and 
approaches  described  in  the  documents  have  changed,  but  the  underlying  reason  for 
having  the  documents  has  remained  constant:  to  capture  the  "best  R&M  practices"  for 
designing  and  producing  reliable  and  maintainable  systems.  "Best  R&M  practices"  refer 
to  those  engineering  and  manufacturing  tasks,  methods,  and  approaches  that  have  proven 
effective  in  fielding  systems  with  good  R&M  performance. 

In  the  past,  commercial  companies  based  their  R&M  programs  on  those  of  the  DoD  and 
adopted  many  DoD  standards,  revising  them  as  necessary  for  their  purpose.  Within  the 
last  few  years,  however,  the  commercial  industries  in  the  United  States  have  undergone 
what  has  been  described  as  chaotic  change.  This  change,  which  continues  today,  is  a 
response  to  an  increasingly  competitive  world  marketplace  in  which  quality  and  R&M 
have  become  the  foundation  of  success.  Unfortunately,  progress  in  quality  and  R&M 
has  been  led,  or  at  least  is  perceived  to  have  been  led,  by  countries  other  than  the  US.  In 
an  attempt  to  "catch  up,"  US  commercial  companies  have  been  aggressively  developing 
or  enhancing  tools  and  methods  for  achieving  high  levels  of  quality  and,  presumably, 
R&M.  However,  DoD’s  general  approach  to  achieving  good  R&M  and  the  DoD 
standards  on  R&M  has  not  changed  substantially  in  over  15  years,  and  may  not  reflect 
the  best  practices  recently  developed  and  currently  used  by  the  commercial  sector. 

R&M  are  not  the  only  disciplines  for  which  the  validity  of  standards  is  in  question. 
Many  top-level  government  officials  are  inclined  to  abolish  the  use  of  (or  require  a  waiver 
to  use)  government-mandated  standards  (and  related  documents  such  as  specifications) 
and  adopt  a  commercial  approach  to  procuring  goods  and  services.  It  is,  in  the  opinion  of 
these  officials,  not  simply  a  question  of  whether  these  documents  are  current,  but 
whether  they  are  or  can  be  applied  effectively.  Too  often,  a  contract  will  include  an 
almost  inexhaustible  list  of  standards,  handbooks,  and  specifications.  Many  of  these  have 
little  if  any  applicability  to  the  product  or  service  involved.  Others  are  cited  without 
"tailoring."  Tailoring  literally  means  making  the  document  fit  the  specific  application.  In 
a  world  of  tight  schedules  where  inexperienced  young  officials  often  develop  contract 
requirements,  lists  of  government  documents  are  often  blindly  added  with  no  tailoring  and 
no  appreciation  of  the  associated  cost. 

1.2  BENCHMARKING 

Benchmarking,  a  term  originally  used  by  land  surveyors  to  compare  elevations,  was 
pioneered  in  the  U.S.  business  community  by  Xerox  Corporation  in  the  late  1970s.  As 
introduced  by  Xerox,  benchmarking  is  a  process  for  finding  the  world-class  standards  for 
a  product,  service,  or  system  and  then  adjusting  one's  own  products,  services,  and 
systems  to  exceed  those  standards^.  These  world-class  standards  can  be  found  by  looking 
at  competitors  who  are  recognized  leaders  for  the  product,  service,  or  system. 
Benchmarking  done  using  competitors  as  models  is  called  competitive  or  strategic 


more  formal  definition  is:  a  systematic  and  rigorous  examination  of  your  organization's  product,  service,  or  work  processes 
measured  against  those  of  organizations  recognized  as  the  best,  to  produce  changes  and  improvements  in  your  enterprise.  (Ettorre, 
Barbara,  “Benchmarking:  The  Next  Generation,”  Management  Review.  June  1993,  pp  11-16) 
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benchmarking'*.  Operating  units  or  functions  within  a  company  can  also  be  used  as  the 
model;  this  approach  is  called  internal  benchmarking.  Finally,  benchmarking  can  be  done 
using  companies  that  are  the  best  practitioners  of  a  particular  function,  regardless  of  what 
industry  the  exemplar  is  in.  This  type  of  benchmarking  is  called  functional  benchmarking. 
Benchmarking  is  a  method  now  commonly  used  by  companies  as  part  of  their  efforts  to 
improve  quality  and  is  a  necessary  element  for  winning  the  Malcolm  Baldrige  National 
Quality  Award  (in  1991,  benchmarking  was  added  by  the  Baldrige  organization  as  an 
element  of  the  Information  and  Analysis  examination  category).  The  model  used  by 
Xerox  for  benchmarking  is  shown  in  Figure  1.  A  similar  model  is  described  in  a  book^  by 
Gregory  H.  Watson,  a  vice-president  of  Xerox. 

Another  form  of  benchmarking  is  called  normative  benchmarking.  In  normative 
benchmarking,  a  consultant  collects  data  form  a  group  of  companies  on  a  product,  service, 
or  process  and  delivers  a  set  of  statistics  to  the  companies  with  the  company  names 
expunged.  The  effort  described  in  this  report  includes  elements  of  normative  and 
functional  benchmarking,  and  the  project  findings  provide  a  reasonable  understanding  of 
the  reliability  practices  used  by  commercial  industry. 


Source:  Geber,  Beverly,  "Benchmarking:  Measuring  Yourself  Against  the  Best,"  Training. 
Nov.  1990,  pp.  36-44. 


Figure  1:  Xerox’s  Ten-Step  Model  for  Benchmarking. 


^Definitions  of  the  types  of  benchmarking  are  taken  from  Strategic  Benchmarking  bv  Gregory  H.  Watson,  John  Wiley  &  Sons,  NY, 
1993. 

^See  Strategic  Benchmarking, ,  Gregory  H.  Watson,  John  Wiley  and  Sons,  1993,  New  York,  NY. 
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1.3  ORGANIZATION  OF  REPORT 

This  report  is  organized  into  five  chapters  and  six  appendixes.  Chapter  1,  Introduction, 
provides  background  information  and  states  the  objective  of  the  study.  Chapter  2  is  a 
discussion  of  the  technical  approach  used  in  performing  the  study.  Results  of  the  study 
are  described  in  Chapter  3.  Conclusions  and  general  recommendations  are  provided  in 
Chapters  4  and  5,  respectively.  Appendix  A  is  a  discussion  of  Defense  Acquisition 
Reform.  Appendix  B  includes  the  detailed  results  of  the  literature  search  and  the  survey 
of  companies.  Appendix  C  contains  an  example  of  the  forms  used  for  the  surveys. 
Appendix  D  contains  the  interview  questions  and  a  copy  of  the  interview  notes.  All 
information  from  the  surveys  and  interviews  have  been  treated  on  a  not-for-attribution 
basis  as  requested  by  the  participating  companies.  The  interview  notes  in  Appendix  D, 
Section  2,  provide  expurgated  summaries  of  the  interviews.  Appendixes  E  and  F  are 
Terms  and  Definitions  and  a  RAC  Product  Order  Form,  respectively. 
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2.0  TECHNICAL  APPROACH 


2.1  OVERVIEW 

A  four-step  approach  was  used.  First,  a  literature  search  was  made  to  identify: 

•  any  documented,  current  commercial  reliability  practices 

•  possible  points  of  contact  in  the  commercial  world 

•  any  previous,  similar  studies 

Next,  a  survey  was  made  of  a  broad  cross-section  of  commercial  companies  representing 
seven  industries.  The  survey  was  made  to  collect  data  on  the  reliability  practices  used  by 
these  companies  to  design  and  produce  reliable,  easily  maintained  products.  As  requested 
by  the  participants,  the  survey  results  were  treated  on  a  non-attribution  basis.  The 
survey  approach  approximated  what  is  termed  normative  benchmarking.  In  normative 
benchmarking,  a  consultant  collects  data  from  a  group  of  companies  on  a  product,  service, 
or  process  and  then  delivers  a  set  of  statistics  to  the  companies  with  company  names 
expunged. 

The  third  step,  interviewing  selected  companies,  was  conducted  to  obtain  more  detailed, 
in-depth  information  on  the  R&M  practices  identified  in  the  survey. 

Finally,  the  literature  search  and  survey  results  were  analyzed  to: 

•  identify  "best"  reliability  practices  by  product  or  system 

•  compare  the  commercial  practices  with  those  documented  in  DoD  standards 
and  handbooks 

2.2  LITERATURE  SEARCH 

A  search  was  made  of  the  RAC  library  and  through  the  Defense  Technical  Information 
Center  (DTIC)  and  the  International  Benchmarking  Clearinghouse^  (BBC).  As  previously 
stated,  the  search  was  conducted  for  three  purposes. 

•  Determine  if  current  commercial  reliability  practices  are  documented  and,  if  so, 
to  identify  them 

•  Identify  possible  points  of  contact  in  the  commercial  world 

•  Determine  if  previous,  similar  studies  had  been  made  and,  if  so,  identify  the 
findings  and  recommendations 


service  of  the  American  Productivity  and  Quality  Center. 
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Searches  were  made  of  the  RAC  library  and  through  DTIC  and  the  IBC  using  the 
following  keywords  in  conjunction  with  the  keywords  reliability  and  maintainability: 
benchmarking,  best  practices,  commercial  practices,  and  Total  Quality  Management 
(TQM).  "Brute-force"  searches  were  also  made  of  the  RAC  library  searching  only  on 
reliability,  then  R&M,  and  finally  on  TQM.  These  searches,  as  would  be  expected, 
resulted  in  hundreds  of  documents  being  identified;  a  manual  review  of  the  titles  and 
abstracts  was  then  made  to  identify  documents  pertinent  to  the  study.  Appendix  B 
contains  the  results  of  the  literature  search. 

2.3  SURVEYS 

To  gather  information  on  the  procedures  being  used  by  commercial  industry  to  develop 
reliable  and  easily  maintained  products,  over  100  companies  representing  seven  different 
industries  (based  on  the  Standard  Industrial  Classification  [SIC]  codes^)  were  identified  as 
candidates  to  participate  in  a  survey.  Based  on  calls  made  to  points  of  contact  (POCs) 
for  these  companies,  companies  were  selected  to  participate  in  the  survey. 

2.3.1  Rationale  for  Surveys 

Ideally,  each  of  the  candidate  companies  would  have  been  visited.  Face-to-face 
discussions  and  tours  of  facilities  are  the  preferred  method  for  gathering  the  type  of 
information  needed  for  this  study.  However,  the  companies  covered  a  geographic  area 
spanning  the  United  States,  from  east  to  west  and  north  to  south.  Budget  and  time 
constraints  did  not  allow  for  on-site  visits  to  all  the  companies.  Also,  it  is  sometimes 
difficult  to  convince  a  company  to  make  time  for  a  visit  unless  some  previous  relationship 
has  been  established  and  the  visit  is  acknowledged  to  be  of  mutual  value. 

Mailed  surveys  are  an  inexpensive  and  relatively  quick  way  of  gathering  such  information. 
However,  surveys  have  two  disadvantages.  First,  the  response  to  surveys  is  historically 
poor;  often  insufficient  data  are  collected  to  support  a  meaningful  evaluation.  Second,  the 
responses  to  a  survey  may  not  be  complete,  may  be  vague,  or  may  be  inappropriate  due 
to  misinterpreting  a  question.  A  means  had  to  be  found  to  capitalize  on  the  advantages  of 
a  mailed  survey  while  avoiding  the  pitfalls. 

A  five-step  approach  was  adopted  for  effectively  using  a  mailed  survey. 

Call  the  POCs  identified  for  all  candidate  companies.  Confirm  that  POCs  are 
the  correct  individuals  to  discuss  their  companies'  reliability  practices.  If  not, 
determine  correct  POCs. 

•  Solicit  participation  of  POCs  on  behalf  of  their  companies  in  the  mailed 
survey  effort.  Explain  the  purpose  of  the  project  and  resulting  product.  Mail 


7 

The  SIC  codes  are  established  and  published  in  the  Standard  Industrial  Classification  Manual  by  the  Executive  Office  of  the 
President,  Office  of  Management  and  Budget. 
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survey  only  to  POCs  "committing"  to  participate  on  behalf  of  their  companies 
(i.e.,  qualify  the  company). 

•  Keep  the  survey  form  short  and  require  least  amount  of  writing  by  the  person 
completing  the  form  (the  final  form  consisted  of  one  page  of  introductory 
material  and  three  pages  with  a  total  of  ten  questions) 

•  Develop  questions  not  subject  to  broad  interpretation  that  focus  on  the  tasks 
(i.e.,  procedures)  used  to  develop  reliable,  maintainable  products. 

•  Make  follow-up  calls  as  needed  based  on  responses  to  survey. 

Determining  the  person  (POC)  for  each  company  with  whom  to  discuss  reliability 
increased  the  probability  of  a  receptive  audience.  Personally  calling  the  person  and 
discussing  the  objective  of  the  project  established  a  level  of  understanding.  These 
precautions  were  taken  to  avoid  the  poor  response  normally  associated  with  "blind 
mailings,"  to  establish  a  relationship  with  the  company,  and  to  reduce  the  possible 
number  of  interpretations  of  the  survey  questions.  Finally,  follow-up  calls  allowed  any 
questionable  responses  to  be  clarified. 

2.3.2  Selection  and  Qualification  of  Companies 

As  was  previously  stated,  seven  key  industries  from  which  to  select  companies  to 
participate  in  the  study  were  identified.  These  industries,  and  their  SIC  codes  are  listed  in 
Table  1.  The  industries  identified  cover  a  broad  range  of  product  types,  product 
complexity  (level  of  technology),  and  include  both  manufacturing  and  service  companies. 

After  identifying  the  industries,  companies  within  these  industries  were  identified  as 
candidates  for  the  survey.  In  identifying  the  candidate  companies,  it  was  important  to  get 
a  good  cross  section  of  products,  not  just  by  type  of  product  but  by  level  of  complexity 
and  rate  of  production.  Most  of  the  weapon  systems  procured  by  DoD  are  complex  and 
produced  in  volumes  that  are  low  in  comparison  to  production  volumes  for  most 
commercial  products.  Even  within  the  DoD,  however,  some  systems  or  products  that  are 
procured  are  not  overly  complex  and  are  bought  in  large  numbers.  So  it  was  important 
that  the  products  manufactured  by  the  candidate  companies  represented  a  reasonably 
broad  range  of  complexity  and  production  volume.  Also,  it  was  important  to  identify 
some  companies  that  do  not  manufacture  products  but  are  large  "consumers"  of  products 
from  other  companies.  These  latter  companies  represent  the  service  and  communications 
industry  listed  in  Table  1.  In  short,  the  range  of  manufacturing  companies  was  to  be 
representative  of  the  range  of  defense  companies,  and  some  companies  that  are 
"comparable"  to  DoD  (i.e.,  they  purchase  products  that  must  be  reliable  and 
maintainable)  had  to  be  chosen.  The  complete  set  of  criteria  for  identifying  candidate 
companies  is  shown  in  Table  2.  Only  one  criterion  had  to  be  met  for  a  company  to  be 
identified  as  a  candidate. 
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Table  1:  Industries  Identified  for  Study 


Industry  Title* 

SIC  Code(s)** 

Examples  of  Products  or  Services 

Measuring,  Controlling  &  Medical 

Equipment 

381-385  and  387 

X-ray  machines,  MRI  equipment,  boiler 
controls,  digital  testers,  flow  meters,  optical 
mirrors,  thermometers 

Transportation  Equipment 

371-379 

Automobiles,  buses,  gasoline  engines, 
motor  vehicle  parts,  aircraft  engines,  ships 

Electrical  &  Electronic 

361-369 

Transformers,  dynamos,  power  regulators, 
microwave  ovens,  washers  and  dryers, 
light  bulbs,  telephone  and  telegraph 
equipment,  television 

Industrial  Machinery  &  Equipment 

351-356 

358-359 

Gas  turbines,  marine  engines,  farm 
elevators,  mining  equipment,  elevators  and 
escalators,  conveyers,  cranes,  machine 
tools,  linotype  machines,  ball  and  roller 
bearings 

Computer  and  Office 

357 

Mini  and  micro  computers,  mainframes, 

PCs,  disk  drives,  calculators,  plotters, 
postage  meters,  and  cash  registers 

Photographic  Equipment 

386 

Cameras,  diaziotype  equipment, 
photographic  film,  photocopy  machines 

Service  &  Communications 

411,  451 

481,  and  491 

Subway  and  light  rail  transport,  air 
transportation,  telephone  service,  electric 
utility  service 

*Abbreviated  version. 

**The  Industry  Group  No.  was  used  rather  than  the  Major  Group  because  the  Major  Groups  are  very  broad.  For  example.  Major 
Group  38  includes  measuring,  analyzing,  and  controlling  instruments,  photographic,  medical,  and  optical  groups;  and  watches  and 
clocks. 


Table  2:  Criteria  for  Identifying  Candidate  Companies 


•  product  is  produced  in  low  quantity  and  is  very  complex  (e.g.,  magnetic  resonance 
imaging  machine),  or 

•  product  is  produced  in  large  quantity  and  is  relatively  simple  in  design  or  to 
manufacture  (e.g.,  PCs),  or 

•  product  is  produced  in  large  quantity  and  is  fairly  complex  in  design  or  to 
manufacture  (e.g.,  automobiles),  or 

■  company  purchases  complex,  expensive  products  (e.g.,  airlines,  utility  companies),  or 

•  company  is  a  Baldrige  Award  winner,  or 

•  company  has  a  reputation  for  or  advertises  high  reliability,  or 

•  company  produces  or  uses  products  for  which  reliability  is  essential  for  safety  (e.g., 
airlines,  nuclear  power  plant,  aircraft  manufacturer),  or 

•  company  is  a  major  buyer  or  integrator  of  systems  for  which  reliability  is  important 

_ (e.g.,  system  integrator) _ 

Using  the  criteria  listed  in  Table  2,  sources  such  as  the  Thomas  Register  were  used  to 
identify  about  15  companies  within  each  industry^.  A  POC(s)  was  then  identified  for 
each  of  these  companies.  To  do  this,  the  Thomas  Register,  the  literature  search,  the 
RAC’s  own  knowledge  base  (i.e.,  previous  work  done  for  a  company  by  the  RAC  or 
IITRI),  and  membership  lists  of  technical  and  professional  groups  (e.g.,  the  Society  of 

^Our  goal  was  to  identify  a  tota!  of  at  least  105  companies  evenly  distributed  across  the  seven  industries,  obtain  the  participation  of 
70  companies  (70%  of  total),  and  to  actually  receive  completed  survey  forms  from  40  companies  (60%  of  participants,  40%  of  total 
identified). 
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Automotive  Engineers  [SAE]  G-11  Committee  on  Reliability,  Maintainability,  and 
Supportability)  were  used.  In  most  cases,  several  possible  POCs  were  identified  for  a 
company,  and  the  "best"  one  could  not  be  determined.  In  those  cases,  one  with  whom 
to  begin  was  arbitrarily  picked. 

In  summary,  qualifying  a  company  consisted  of  ensuring  it  met  one  of  the  criteria  in  Table 
2,  determining  the  "best"  person  in  that  company  with  whom  to  discuss  the  survey,  and 
obtaining  a  "commitment"  to  participate  in  the  survey.  The  "best  person"  was  someone 
who  worked  in  the  area  of  reliability  and  was  in  a  position  that  provided  visibility  into 
the  company's  overall  reliability  policy.  A  commitment  was  a  statement  by  the  POC  that 
the  survey  form  would  be  completed  and  returned  within  two  weeks  of  receipt. 

One  lesson  learned  in  calling  POCs  was  that  the  process  is  time-consuming  and  requires  a 
great  deal  of  patience  and  persistence.  More  than  300  calls  were  needed  to  contact  the 
right  POC  for  each  of  the  more  than  100  candidate  companies.  Individuals  were  on  travel, 
in  meetings,  or  no  longer  with  the  company  (even  individuals  who  were  with  the 
company  within  the  previous  few  months  had  left  or  had  transferred  to  another  division). 
In  today's  volatile  economic  environment,  reorganizations  and  downsizing  are  becoming 
the  norm;  our  efforts  to  contact  individuals  were  hampered  by  this  reality.  In  addition,  at 
least  one  of  the  companies  believed  to  be  a  manufacturing  company  had  recently  divested 
itself  of  all  manufacturing  and  is  now  a  systems  integration  firm.  (Incidentally,  this  firm 
was  included  because  it,  like  DoD,  has  the  task  of  allocating  system-level  requirements  to 
the  subsystem  level,  contracting  with  suppliers  to  provide  the  subsystems,  and  then 
verifying  that  each  subsystem  meets  the  allocated  requirements.)  Finally,  some 
companies  declined  to  participate  due  to  concerns  for  proprietary  data^,  due  to  the  costs 
associated  with  the  time  to  complete  the  survey,  and  simply  because  they  were  not 
interested. 

2.3.3  Development  of  Survey  Form 

The  survey  form  consisted  of  introductory  material  and  a  set  of  questions.  Two  versions 
of  the  form  were  developed:  one  for  service  firms  and  one  for  firms  manufacturing 
products  (copies  of  each  are  provided  in  Appendix  C).  The  introductory  material  of  the 
survey  provides  a  common  basis  of  understanding:  the  purpose  of  the  project,  purpose  of 
the  survey,  and  definition  of  reliability  tasks.  The  questions  were  designed  to  determine: 

•  what  reliability  tasks  are  used 

•  when  during  the  product  life  cycle  the  tasks  are  performed 

•  the  relative  importance  and  effectiveness  of  the  reliability  tasks 

•  the  use  of  non-DoD  reliability  standards  and  handbooks 


^We  explained  that  information  would  be  handled  on  a  non-attribution  basis  and  treated  as  proprietary.  We  also  explained  that 
such  information  would  not  be  releasable  under  the  Freedom  of  Information  Act  (another  concern  voiced  by  some).  Finally,  we 
offered  to  sign  non-disclosure  agreements.  Despite  these  efforts,  some  individuals  and  companies  declined  to  participate. 
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•  the  source  of  reliability  task  requirements  (i.e.,  customer,  company  policy, 
government  regulations,  etc.) 

•  the  nature  of  the  product  (i.e.,  type,  production  volume,  unit  cost,  etc. 

2.4  INTERVIEWS 

The  third  step  was  to  interview  a  small  number  of  companies  (for  practical  and  funding 
considerations,  the  number  of  companies  to  be  interviewed  was  limited  to  15)  to  obtain 
more  detailed  information  on  the  reliability^'^  practices  being  used.  Beginning  in  October 
1993,  the  project  team  began  contacting  companies  to  schedule  interviews.  The 
companies  targeted  for  interviews  primarily  consisted  of  those  companies  that 
participated  in  the  mailed  survey  but  also  included  a  few  that  did  not. 

Companies  were  chosen  using  the  same  criteria  used  to  select  companies  for  the  mailed 
survey.  In  addition,  companies  that  had  provided  the  most  promising”  responses  to  the 
survey  were  asked  to  participate  in  the  interviews.  These  criteria  helped  to  focus  on 
those  companies  that,  on  the  basis  of  their  responses  to  the  survey,  appeared  to  have  new 
procedures  or  unique  applications  of  existing  reliability  procedures. 

A  set  of  interview  questions  was  developed  (see  Appendix  D).  The  questions  were  used 
as  a  guide  and  to  ensure  that  a  common  set  of  questions  was  asked  of  each  interviewee. 
However,  the  interviews  were  not  restricted  to  only  these  questions  or  the  order  in  which 
they  were  written;  instead,  the  interviewers  followed  the  lead  of  those  being  interviewed 
and  allow  the  discussion  to  take  its  course.  Two  RAC  staff  members  participated  in  each 
interview  to  ensure  that  a  complete  and  accurate  record  of  the  interview  was  taken  and  to 
foster  a  more  productive  discussion.  Two-hour  interviews  were  scheduled  at  the 
convenience  of  the  company  and  individual  being  interviewed.  In  some  cases,  an 
individual  or  company  declined  to  be  interviewed.  Some  of  the  reasons  given  for  declining 
to  be  interviewed  are  the  same  as  those  given  for  not  participating  in  the  mailed  survey: 
proprietary  data,  cost,  and,  simply,  no  interest.  The  names  of  the  companies  that 
participated  in  the  interviews  are  listed  in  Table  3. 


^'^Subsequent  to  the  literature  search  and  surveys,  the  project  was  rescoped  to  focus  on  reliability. 

”ln  other  words,  responses  that  indicated  a  new,  possibly  revolutionary  R&M  procedure  not  incorporated  in  DoD  standards  and  handbooks, 
or  a  unique  or  highly  successful  application  of  an  established  procedure. 
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Table  3:  List  of  Companies  Interviewed 


Company 

Products  Include 

*AC  Rochester 

Flint,  MI 

Engine  management  systems 

Bell  Helicopter 

Fort  Worth,  TX 

Rotary-wing  aircraft 

*Boeing  Commercial 

Airplane  Group 

Renton,  WA 

Commercial  aircraft 

♦Carrier  Corp. 

Syracuse,  NY 

Commercial  and  industrial  cooling  systems 

♦Eastman  Kodak 

Rochester,  NY 

Film,  blood  analyzers,  photocopy  machines 

♦ELDECCorp. 

Lynnwood,  WA 

Power  conversion,  monitor  and  control,  and 
aircraft  sensing  systems 

General  Motors  Corp. 

Midsize  Car  Div. 

Warren,  MI 

Automobile  platforms  and  components 

Intermagnetics  General  Corp. 
Guilderland,  NY 

Magnets  for  MRIs 

Motorola  Cellular  Phone 
Libertyville,  IL 

Cellular  phones 

Ohmeda 

Columbia,  MD 

Incubators,  Bili-blankets,  infant  warmers 

♦Sun  Microsystems 

Computer  Corp. 

Mountain  View,  CA 

Microcomputers 

♦Participated  in  mailed  survey 
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3.0  RESULTS 


3.1  LITERATURE  SEARCH  RESULTS 

A  total  of  40  documents  were  identified  and  collected  in  support  of  this  study.  A 
complete  list  of  the  documents  is  provided  in  Appendix  B,  as  are  synopses  of  those 
documents  most  relevant  to  the  study. 

3.2  SURVEY  RESULTS 

Of  the  more  than  300  POCs  (representing  over  100  candidate  companies)  contacted,  78 
(representing  73  companies)  agreed  to  participate.  Of  these,  40  provided  responses. 
Edited  versions  of  their  responses and  a  copy  of  the  survey  forms  are  included  in 
Appendix  B.  Table  4  provides  a  summary  of  the  companies  interviewed  and  surveyed,  in 
terms  of  the  product  markets,  technologies,  production  volume,  unit  cost,  and  type  of 
product.  (The  nature  of  new  development  was  not  revealed  through  the  surveys  but 
from  the  interviews.) 

Table  4:  Overview  of  Companies  Surveyed  and  Interviewed 


Criteria 

Description  for  Companies  Surveyed  and  Interviewed 

Types  of  Products 

Automobiles  and  automotive  products,  telephones  and  test  equipment, 
heating  and  air  conditioning  equipment  and  systems,  diesel  engines, 
computer  workstations,  data  communication  products,  aircraft  fuel  and 
speed  controls,  commercial  and  general  aviation  aircraft,  diesel/electric 
locomotives,  medical  equipment 

Nature  of  New 
Development 

Predominately  evolutionaiy  in  nature.  Revolutionary  changes  or 
improvements  in  products  are  the  exception.  Proven  design  and 
technology  characterizes  most  development. 

Unit  Cost 

Ranges  from  less  than  $100  per  unit  to  several  millions  of  dollars  per  unit 

Production  Volume 

Ranges  from  a  single  unit  per  year  (large,  custom  product)  to  millions  of 
units  per  year 

Technologies 

Range  from  proven,  off-the-shelf  technologies  to  state-of-the-art 
technologies  for  material,  processes,  and  functional  design. 

Markets 

Include  mass  markets  in  the  U.S.;  world-wide  markets;  small,  select 
customer  markets;  and  all  level  of  customers  (i.e.,  industry,  OEMs, 
general  public,  etc.) 

The  survey  proved  to  be  most  useful  in  identifying  specific  tasks  that  are  performed, 
when  they  are  performed,  why  they  are  performed  (i.e.,  required  by  customer  or  required 
by  company),  and  which  tasks  are  most  important.  Also,  the  surveys  provided  some 
insight  into  the  guidance  and  regulatory  documents  used  or  imposed  on  commercial 
industries. 


^^Most  responses  were  hand  written.  Editing  was  performed  to  eorrect  spelling,  delete  the  names  of  eompanies  to  treat  the 
responses  on  a  non- attribution  basis,  and  to  reflect  clarifications  obtained  in  follow-up  calls.  The  edited  versions  were  then  typed  to 
make  them  easier  to  read. 
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One  question  in  the  survey  form  asks  for  information  on  the  reliability  tasks  being 
performed  by  the  company:  what  tasks  are  performed,  at  what  phase  of  the  product's  life 
they  are  performed,  and  what  is  the  relative  importance  of  each  task.  Another  question 
asks  if  the  company  tries  to  determine  the  value  of  specific  tasks.  Although  many 
respondents  answered  yes  to  the  latter  question,  only  one  indicated  a  method  that  would 
allow  the  value  of  a  specific  task  to  be  judged  (the  probable  cost  of  failure  and  the  task 
cost  are  compared). 

Based  on  the  responses  to  the  questions  dealing  with  tasks,  the  "most  important" 
reliability  tasks  were  identified  in  three  ways.  First,  those  tasks  that  were  most 
frequently  cited  (i.e.,  what  percent  of  respondents  said  the  task  was  used)  were  identified. 
Second,  those  with  the  highest  normalized  scores  (based  on  importance  rating  given  by 
respondents)  were  identified.  Finally,  those  tasks  ranked  2  or  3  in  importance  by  the 
respondents  were  given  a  positive  rating;  and  those  ranked  0  or  1  were  given  a  negative 
rating.  Table  5  and  Figure  2  summarize  the  results  of  these  three  perspectives  of 
importance  (we  also  considered  average  rankings,  tasks  receiving  highest  percentage  of  a  3 

Table  5:  Most  Important  Reliability  Tasks  According  to  Survey 


A.  Most  frequently  cited  tasks  and  when  they  are  performed 


Rank 

Task 

%  of  Respondents  Who 
Cite  the  Tasks* 

When  the  Task  is  Performed** 

Development 

Production 

1 

FRACAS 

97 

51 

65 

2 

Design  Reviews 

89 

91 

24 

3 

Predictions 

89 

91 

12 

4 

FMECA 

86 

90 

20 

5 

RQT 

86 

89 

32 

6 

Sub/Vendor  Control 

84 

70 

76 

7 

Parts  Control 

84 

63 

72 

8 

ESS 

78 

66 

55 

9 

1  Thermal  Analysis 

73 

86 

14 

10 

TAAF 

73 

90 

50 

B.  Most  important  tasks  based  on  normalized  score 


Rank 

Task 

Normalized  Score*** 

1 

FRACAS 

88.3 

2 

Design  Reviews 

83.8 

3 

SubWendor  Control 

72.1 

4 

Parts  Control 

71.2 

5 

FMECA 

68.5 

6 

RQT 

70.3 

7 

Predictions 

62.2 

8 

TAAF 

59.5 

9 

Thermal  Analysis 

58.6 

10 

ESS 

54.1 

*And  who  rate  it  as  1,  2,  or  3  in  importance  (not  all  respondents  rated  importance) 
**  Percent  of  those  citing  the  task 

♦♦♦Normalized  score  =  100%  x  (Z  importance  entries )/(number  of  entries  *  3). 
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ranking,  and  tasks  receiving  highest  percentage  of  a  2  or  3  ranking  in  making  an  overall 
evaluation  of  importance).  Note  that  although  the  task  titles  are  those  used  by  DoD,  the 
task  implementation  within  the  commercial  sector  may  be  quite  different  from  that  in  a 
military  standard.  Evidence  of  this  difference  was  found  during  follow-up  calls  to  survey 
participants  and  from  interviews. 


40 


ABCDEFGH  IJKLMNOPQR 


A  =  FMEA,  B  =  FT  A,  C  =  Thermal  Analysis,  D  =  Sneak  Circuit  Analysis,  E  =  Worst-Case  Circuit  Analysis,  ] 
=  Parts  Control,  G  =  Vendor  Control,  H  =  Derating,  I  =  Predictions,  J  =  Allocations,  K  =  Modeling,  L  -  Desil 
Review,  M  =  Failure  Reporting  and  Corrective  Action  System,  N  =  Environmental  Stress  Screening,  O  ~  Te^ 
Analyze,  and  Fix,  P  ==  Reliailbity  Growth  Testing,  Q  =  Reliability  Qualification  Testing,  R  =  Maintainability 
Demonstration 


Figure  2:  Relative  importance  of  reliability  tasks  (number  of  respondents  rating  a 
task  2  or  3  [positive  values]  or  0  (no  rating)  or  1  [negative  values]) 

Regulatory  documents  imposed  or  used  by  the  respondents  included  Federal 
Communications  Commission  (FCC)  and  Department  of  Transportation  (DOT) 
regulations,  American  Society  of  Mechanical  Engineers  (ASME)  specifications. 
Occupational  Safety  and  Health  Administration  (OSHA)  regulations.  Environmental 
Protection  Agency  (EPA)  emissions  standards,  DoD  and  U.S.  Military  Specifications  and 
standards.  Underwriters  Laboratory  (UL)  requirements,  and  Canadian  Standards 
Association  (CSA)  and  International  Standards  Organization  (ISO)  standards. 
Documents  used  to  define  procedures  for  Reliability  tasks  include  MIL-HDBK-217, 
Bellcore  TR-NWT-00332,  National  Aeronautics  and  Space  Administration  (NASA)- 
STD-978,  MIL-STD-2155,  Nuclear  Regulatory  (NUREG)  Fault  Tree  Analysis  (FTA) 
and  Computer-aided  FTA  (CAFTA),  MIL-STD-1629,  Federal  Aviation  Administration 
(FAA)  Circular  120-17A,  MIL-STD-883,  and  Association  for  the  Advancement  of 
Medical  Instrumentation  (AAMI)  STD  for  Pacemakers. 
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Additional  insight  into  the  meaning  and  limitations  of  the  survey  responses,  especially  in 
evaluating  specific  tasks,  was  gained  from  the  interviews  that  were  conducted. 

3.3  INTERVIEW  RESULTS 

Complete  notes  were  taken  during  each  interview.  In  Appendix  D,  Section  2,  are 
summaries  of  all  interviews!^.  One  insight  gained  from  the  interviews  that  was  not 
available  from  the  survey  was  the  nature  of  product  development.  In  general,  the 
products  built  by  the  companies  are  evolutionary,  rather  than  revolutionary  in  nature. 
Although  new  technologies  are  found  in  many  of  these  products,  each  successive  product 
is  an  improvement  or  advancement  of  its  predecessor.  Even  the  new  technologies  tend  to 
be  adopted  from  other,  more  risky  segments  of  industry  (e.g.,  the  defense  industry  and 
basic  R&D  organizations).  Also,  a  product  is  not  pushed  to  market  before  extensive 
testing  is  done  and  the  design  and  risks  are  well-understood. 

An  analysis  of  the  interviews  showed  that  the  information  provided  in  the  surveys  must 
be  viewed  with  some  reservations  for  three  reasons. 

1 .  It  was  found  that  although  certain  reliability  tasks  might  have  been  cited  in  the 
survey  as  being  important,  the  company  is  not  performing  the  tasks  as  part  of 
the  product  development  efforts.  In  other  words,  a  survey  response  that  a 
task  is  performed  actually  could  mean  "I  (the  respondent)  think  it  is 
important." 

2.  Closely  related  to  the  previous  point  is  many  of  those  interviewed  indicated 
that  their  responses  represent  their  company's  vision  for  the  future;  some  of 
this  vision  may  not  yet  be  implemented.  Consequently,  some  stated  “best 
practices”  are  either  not  fully  implemented  or  are  immature  and  the  tasks 
identified  in  the  survey  as  important  may  not  yet  be  in  use. 

3.  Information  from  the  interviews  contradicts  information  from  the  surveys. 
For  example,  qualification  testing  received  a  normalized  score  of  70.3  based  on 
the  survey  responses.  Yet,  in  the  interviews,  even  those  of  survey 
respondents,  qualification  testing  was  generally  criticized  as  being  of  no  value. 
It  is  most  probable  that  the  survey  questions  were  not  accurately  stated,  the 
possible  responses  were  too  limited,  or  survey  respondents  were  then  defining 
the  tasks  differently  (despite  the  fact  that  definitions  were  included  with  the 
survey  forms). 

The  remaining  discussion  of  the  findings  from  the  interviews  will  be  covered  in  the 
following  manner.  First,  an  overall  discussion  of  the  interview  findings  is  provided.  This 
overall  discussion  is  then  broken  out  by  general  and  specific  findings.  A  series  of  general 
findings  is  provided,  followed  by  the  specific  findings  presented  in  the  format  that 
follows. 


!^In  accordance  with  the  wishes  of  the  participants,  only  expurgated  summaries  are  provided.  In  this  way,  specific  comments  or 
views  can  not  be  associated  with  the  contributor. 
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Approach:  [The  composite  approach,  or  philosophy,  articulated  by  the 
companies] 

Implementation:  [The  way  the  companies  apply  the  approach  or  philosophy.] 

3.4  OVERALL  DISCUSSION 

Those  interviewed  agreed  on  several  basic  points  regarding  developing  and  producing  a 
reliable  product.  First,  concurrent  engineering,  also  referred  to  in  terms  such  as  "the 
product  development  team  concept,"  is  essential  to  being  competitive.  Product  teams 
promote  a  feeling  of  "ownership"  by  all  involved  in  the  analyses,  assumptions,  and 
compromises  made  during  the  development  process.  Concurrent  engineering  requires  that 
modern  development  tools,  such  as  CAD/CAM,  be  available,  reliability  analyses  tools 
must  be  a  part  of  the  CAD/CAM  system  so  that  the  reliability  engineer  can  interact  with 
the  development  team  in  real-time.  Analyses  done  "off-line"  by  a  traditional,  separate 
reliability  office  typically  are  completed  too  late  to  affect  the  design.  (Figure  3  illustrates 
the  extent  to  which  CAD/CAM  is  being  implemented  by  some  companies.) 
Unfortunately,  many  of  those  interviewed  admitted  that  reliability  analyses  are  not  yet 
fully  integrated  into  their  CAD/CAM  systems. 


Relational 
data  base 
of  3700  most- 
used  parts 


Optimized  set 
of  product  functions 

Windows  with 
selection  nnenus 

-Qi 


pecision  tablq 
listing 
consistent, 
market- 
optimized 
solutions 


Customer  sales 
Configuration  engineer 


Automatic 
sales/order/manufacturing 
documentation 


Figure  3:  CAD/CAM  facilitates  customer  focus. 


A  second  basic  point  subscribed  to  by  all  those  interviewed  is  the  need  to  understand  the 
physics  of  failure^^^.  Thoroughly  understanding  the  design  is  essential  not  only  to 


^^They  did  not  suggest  that  physics  of  failure  be  a  substitute  for  predictions;  only  that  it  be  a  means  for  understanding  the  design. 
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reliability  but  to  the  entire  range  of  performance  and  "ility"  characteristics.  In  trying  to 
understand  the  design,  however,  it  was  indicated  that  the  limitations  of  deterministic 
design  have  been  the  source  of  problems  in  the  past.  Probabilistic  methods  are  necessary 
to  account  for  unknowns  and  as  a  means  of  developing  a  robust  design.  A  "dynamic 
tension"  between  the  engineer  (deterministic)  and  statistician  (probabilistic)  was  described 
as  a  desirable  state  of  affairs.  It  was  also  pointed  out  that  modern  systems  and 
components  often  have  reliability  requirements  that  cannot  be  measured  in  any  practical 
way  using  currently  available  techniques.  Bayesian  statistical  methods  and  accelerated 
testing  were  methods  mentioned  for  this  purpose. 

Another  commonly  held  view  is  that  the  reliability  of  the  product  is  no  more  important 
than  the  reliability  of  the  processes.  Although  the  subject  of  process  reliability  (including 
the  reliability  of  the  tools  and  manufacturing  equipment)  is  new  to  the  companies 
interviewed,  all  recognize  its  importance. 

Third,  the  interviewees  agreed  on  the  importance  of  strong  supplier  relationships^ ^ 
Those  interviewed  indicated  that  whatever  requirements  were  imposed  within  their 
companiesi^  are  imposed  on  their  suppliers.  A  stable  group  of  suppliers  is  part  of 
commercial  industry’s  method  of  reducing  risk  and  is  consistent  with  a  focus  on  using 
proven  designs. 

Fourth,  customers  are  not  deeply  involved  in  the  design  and  development  process  and,  in 
general,  they  do  not  wish  to  be.  Customers,  in  some  cases,  may  be  asked  to  participate  in 
one  or  more  design  reviews.  Such  participation  is  almost  always  at  the  initiation  of  the 
product  developer  who  want  to  ensure  that  the  voice  of  the  customer  has  been  heard  and 
properly  understood.  Customers  demand  good  products  and  depend  on  competition  and 
market  pressures  to  incentivize  the  contractor  to  make  the  "best  effort"  in  developing  the 
product.  When  service  problems  arise,  successful  contractors  work  with  the  customer  to 
resolve  the  problem  as  quickly  as  possible.  However,  after-sales  problems  can  reduce  a 
company's  profit,  so  considerable  emphasis  is  placed  on  up-front,  concurrent  engineering. 

Finally,  it  is  clear  that  those  interviewed  find  little  fault  in  the  technical  content  of 
military  specifications,  handbooks,  and  standards.  They  suggested  that  the  technical 
information,  such  as  recommended  temperature  dwell  and  cycle  times  during  temperature 
cycling  tests,  in  these  documents  is  of  value. 

Table  6  lists  anecdotal  comments  from  the  interviews  that  provides  additional  insight  into 
the  commercial  perspectives  on  reliability.  In  some  cases,  comments  have  been 
paraphrased  for  the  sake  of  clarity  and  brevity. 

further  indication  of  commercial  industry’s  emphasis  on  supplier  quality  was  reported  in  the  January  1994  issue  of  Quality 
Progress.  According  to  the  article,  Chrysler,  Ford,  and  General  Motors  have  merged  their  standards  for  supplier  quality  systems,  as 
have  major  manufacturers  (e.g.,  Navistar,  International,  and  Freightliner)  of  trucks.  In  so  doing,  the  two  industries  have  established 
a  quality  system  standard  for  suppliers  covering  the  entire  domestic  automotive  industry. 

^^The  scope  of  tasks  and  requirements  required  by  company  policy  varies  greatly  among  the  companies.  Some  have  no  mandates; 
the  product  managers  are  free  to  use  any  or  no  tasks,  and  no  quantitative  requirements  are  developed.  Others  mandate  some  basic, 
general  requirements  (i.e.,  no  single-point  safety  failures)  but  still  vest  most  of  the  authority  and  decision-making  in  the  product 
managers.  To  "encourage"  decisions  that  take  long-term  effects  into  consideration,  the  product  managers  usually  have  life-cycle 
responsibility  for  their  products.  In  one  case,  the  product  manager  must  budget  for  warranty  costs  and  is  held  responsible  for  in- 
service  performance. 
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Table  6.  Anecdotal  Comments 


A  failure  is  any  incident  that  sends  the  customer  back  to  the  dealer  for  repairs. 

Most  critical  challenge  is  converting  the  voice  of  the  customer  into  a  product  specification  (i.e., 
quality  function  deployment). 

CAD/CAM  is  a  tool  for  translating  art  to  part. 

The  quality  of  warranty  data  is  suspect.  Such  data  must  be  supplemented  with  controlled 
testing  and  data  collection  efforts,  and  sensitivity  analyses  used  to  determine  the  "critical  few" 
problems. 

Reliability  is  the  engineering  discipline  with  the  greatest  potential  for  affecting  product  life  cycle 
cost. 

Military  specifications  and  standards,  if  used  blindly  with  no  regard  for  their  applicability  or 
value-added  (i.e.,  no  tailoring  or  applied  too  late  to  influence  the  design),  can  drive  up  costs 
with  no  positive  affect  on  reliability. 

The  military  standards,  in  many  cases,  form  the  basis  for  commercial  reliability  practices.  Their 
limitations  are  out-of-date  information  (i.e.,  not  kept  up-to-date)  and  limited  applicability  of 
data  (i.e.,  MIL-HDBK-217).  For  these  reasons,  many  companies  revise  or  modify  the  standards 
for  their  use  or  develop  their  own  standards  using  the  military  standards  as  a  model. 

Administration  of  MIL-SPECs/STDs/HDBKs,  without  tailoring,  can  impose  documentary  and 
proof  burdens  not  relevant  to  the  close  supplier-contractor  partnerships  seen  in  commercial  business. 

In  commercial  business,  the  contracting  community  clearly  plays  a  support  role  to  the  technical 
community. 

Authority  for  decisions  that  have  long-term  implications  must  be  balanced  by  responsibility  for 
those  implications. 

An  effective  product  design  effort  does  not  compartmentalize  reliability  and  remove  it  from  the 
mainstream  design  effort. 

Demonstration  testing  (i.e.,  MIL-STD-781  tests),  is  an  after-the-fact  activity  that  adds  little 
value;  it  should  be  done  only  if  required  by  the  customer.  Development  testing  (e.g.,  growth 
testing,  fatigue  testing,  proof  testing,  etc.)  should  be  the  focus  of  the  testing  community. 

High  reliability  and  compressed  development  schedules  can  coexist.  Compressed  schedules  force 
the  development  and  use  of  tools  and  methods  that  allow  a  better  understanding  of  the  design, 
an  understanding  that  results  in  higher  reliability. 

Customers  must  recognize  the  need  for  trust  and  partnership  between  themselves  and  their 
contractors.  Contractors  are  incentivized  by  emphasizing  and  nurturing  capital  enterprise. 

Whenever  bureaucracy  and  administrative  burdens  are  imposed  on  a  product  development,  the 
design  and  manufacturing  processes  suffer,  costs  increase,  and  competitiveness  wanes. 

The  most  stmctured  and  cohesive  approach  to  reliability  is  that  used  by  the  military. 

However,  the  approach  used  by  commercial  industry  is  more  economical  and  about  as  effective. 
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3.5  GENERAL  FINDINGS 
3.5.1  The  Environment 

a.  Companies  are  changing  many  of  their  processes  to  incorporate  robust  design, 
concurrent  engineering,  and  other  concepts  to  improve  their  products  and  be 
more  competitive.  Through  these  changes,  reliability  is  being  made  an  integral 
part  of  design,  rather  than  an  “add-on.” 

b.  Companies  are  adopting  a  total  quality  management  approach  to  design  and 
production.  Quality  and  reliability  are  intimately  related  in  the  minds  of  those 
interviewed.  Indeed,  many  companies  have  one  manager  in  charge  of  both 
areas.  Basically,  quality  is  considered  the  more  general  term,  describing  the 
degree  to  which  the  product  satisfies  the  customer’s  needs  and  expectations. 
Reliability  is  just  one  characteristic  that  determines  overall  quality. 

c.  Many  of  those  interviewed  indicated  that  their  responses  represent  their 
company's  vision  for  the  future;  some  of  this  vision  may  not  yet  be 
implemented.  Consequently,  some  stated  “best  practices”  are  either  not  fully 
implemented  or  are  immature. 

d.  The  extent  to  which  customers  are  involved  in  the  design  and  development 
process  varies  but  is  basically  “hands  off’.  Customers  do  not  usually  require 
specific  reliability  tasks  and  analysis  and  often  do  not  explicitly  state  a 
reliability  requirement;  when  they  do,  the  requirement  is  stated  in  functional 
and  general  terms.  These  functional  customer  requirements  are  most  analogous 
to  the  ideal  "user  needs"  discussed  in  DoD  Directive  5000.1.  The  commercial 
contractor  has  a  great  deal  of  flexibility  in  determining  the  design  parameters 
(type  and  magnitude).  Seldom,  if  ever,  is  the  customer  aware  of  the 
contractor's  procedures,  use  of  concurrent  engineering,  or  development  testing 
program.  The  customer  is  only  interested  in  the  results.  The  only  “proof’ 
demanded  by  the  customer  is  the  product’s  reliability  in  actual  use. 

e.  Genuinely  “custom”  products  are  the  exception.  Customizing  a  product  for  a 
customer  usually  involves  “bundling”  different  options  or  components  to  meet 
a  specific  customer’s  needs.  Genuinely  custom,  one-of-a-kind  products 
present  a  higher  risk  to  achieving  high  reliability;  however,  the  approach  of 
using  product  design  teams,  development  testing,  and  proven  designs  still 
applies. 
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3.5.2  The  Practices 

a.  In  most  cases,  commercial  companies  see  no  value  in  reliability  demonstration 
tests,  of  which  qualification  testing  is  one  type.  First,  such  tests  come  much 
too  late  to  make  cost-effective  design  changes.  Second,  the  very  high 
reliability  demanded  of,  and  achieved  in,  modern  designs  cannot  be 
demonstrated  with  any  reasonable  degree  of  statistical  confidence.  Finally, 
these  tests  are  considered  accounting  tests  (as  they  are  called  in  MIL-STD- 
781),  meant  only  to  determine  compliance.  In  contrast,  development  testing, 
intended  to  confirm  the  soundness  of  design,  receives  great  attention. 
Development  testing,  analytical  methods,  and  accelerated  life  testing 
techniques  replace  demonstrations  and  qualification  tests  in  the  commercial 
world. 

b.  Although  the  design  of  products  by  most  of  the  companies  interviewed  is 
evolutionary  in  nature,  some  designs  incorporate  new  technologies  or 
otherwise  reflect  more  of  a  revolutionary  nature.  Between  the  evolutionary 
and  revolutionary  designs,  the  most  frequently  stated  difference  in  the 
technical  approach  was  the  amount  of  development  testing  conducted. 
Whenever  risk  is  higher,  as  is  the  case  with  new  technologies  and  unproved 
designs,  commercial  companies  invest  more  time  and  resources  into  testing.  In 
addition,  more  detailed  and  extensive  analyses,  such  as  a  Failure  Modes  and 
Effects  Analysis  (FMEA),  will  be  conducted. 

c.  Commercial  companies  specify  the  environmental  conditions  for  their 
products.  In  doing  so,  and  for  many  other  reliability  activities,  those 
interviewed  use  military  standards,  although  the  standards  are  often  revised  or 
modified.  They  find  little  fault  in  the  technical  content  of  military 
specifications,  handbooks,  and  standards.  (However,  it  was  stated  that  in 
many  documents  the  data  are  not  current  and  not  always  applicable.  In  the 
case  of  MIL-HDBK-217,  for  example,  some  commercial  companies  have 
developed  their  own  failure  rate  data.)  They  suggested  that  the  technical 
information,  such  as  recommended  temperature  dwell  and  cycle  times  during 
temperature  cycling  tests,  in  these  documents  is  not  duplicated  elsewhere. 

d.  Only  DoD  is  doing  and  funding  the  research  needed  to  develop  technical 
information  included  in  military  standards  and  making  that  information 
available  to  industryi"^.  If  and  when  commercial  companies  perform  such 
research,  the  results  are  usually  proprietary.  So  it  is  not  the  technical  aspect 
of  the  military  documents  that  is  at  fault.  The  fault  is  found  in  the 
enforcement  and  administrative  aspects  that  have  no  value  in  the  commercial 
world.  When  a  document  is  invoked  on  a  defense  contract,  it  is  the  lack  of 


the  time  this  study  was  being  conducted,  a  cooperative  effort  among  the  industrial  societies  was  being  proposed  by  the  Society 
of  Automotive  Engineers.  Under  this  proposed  effort,  the  various  societies  and  representatives  of  industry,  academia,  and 
government  would  work  together  to  develop  “world-class”  reliability,  maintainability,  and  supportability  standards  for  industry. 
These  industry  standards  could  be  cited  in  military  procurements. 
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tailoring,  administration,  and  requirement  for  extensive  documentation  that 
those  interviewed  believe  adds  no  value.  All  cautioned  against  "throwing  out" 
the  military  specifications,  standards,  and  handbooks. 

3.6  SPECIFIC  FINDINGS 

First  the  composite  approach,  or  philosophy,  articulated  by  the  companies  is  given. 

Then,  the  manner  in  which  the  companies  apply  the  approach  or  philosophy  is  described. 

1.  Approach:  Thoroughly  understanding  the  design  is  essential  not  only  to 
reliability  but  to  the  entire  range  of  performance  and  "ility"  characteristics. 

Application:  Extensive  emphasis  is  placed  on  development  testing  to 
characterize  failures  mechanisms,  validate  models  and  analyses,  and  verify 
design  concepts.  FMEA,  physics  of  failure,  and  other  analyses  are  used  to 
identify  critical  areas.  Design  and  test  efforts  focus  on  these  critical  areas.  In 
addition,  all  the  companies  interviewed  make  some  attempt  to  collect  and 
analyze  failure  data.  For  fielded  products,  these  data  are  collected  usually  as 
part  of  the  warranty  program.  Based  on  the  results  of  analyzing  failure  data, 
the  design  (including  parts  selection  and  application),  test  procedures,  or 
manufacturing  processes  may  be  changed.  This  process  is  equivalent  to  the 
failure  reporting  and  corrective  action  system  (FRACAS)  usually  imposed 
under  MIL-STD-785. 

2.  Approach:  The  value  of  testing  is  in  confirming  the  design  and  engineering 
approach,  not  in  showing  compliance.  Customers  almost  never  ask  for 
demonstration  tests;  however,  acceptance  tests,  especially  for  high-value, 
build-to-order  systems  are  sometimes  required  by  the  customer.  Customers 
demand  good  products  and  depend  on  competition  and  market  pressures  to 
incentivize  the  contractor  to  make  the  "best  effort"  in  developing  the  product. 
When  service  problems  arise,  successful  contractors  work  with  the  customer 
to  resolve  the  problem  as  quickly  as  possible.  However,  after-sales  problems 
can  reduce  a  company's  profit,  so  considerable  emphasis  is  placed  on  up-front, 
concurrent  engineering. 

Application:  Demonstration  testing  (also  called  qualification  testing)  is  not 
done,  unless  required  by  the  customer  (seldom)  or  by  a  government  regulation 
(occasionally).  Development,  accelerated,  and  other  engineering  tests  are 
planned  to  characterize  failure  mechanisms;  address  risk  areas;  address  areas  of 
new  technology  or  technology  application;  and  confirm  the  design,  engineering 
models,  and  analyses  results. 

3.  Approach:  Deterministic  design  has  limitations  that  have  been  the  source  of 
problems  in  the  past.  Probabilistic  methods  are  needed  to  account  for 
unknowns  and  as  a  means  of  developing  a  robust  design.  The  high  reliability 
being  demanded  of  and  achieved  in  modern  designs  cannot  be  cost-effectively 
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demonstrated  or  measured  with  any  reasonable  degree  of  statistical  confidence. 
Simulation  and  modeling  play  an  increasingly  important  role  in  compressing 
the  time  schedule  for  the  development  and  characterization  of  new  designs. 

Application;  Accelerated  testing  techniques  (generally  at  the  card-  and  box- 
level),  the  use  of  Bayesian  statistical  methods,  and  other  analytical  methods 
are  used  extensively  to  supplement  deterministic  design.  These  techniques 
and  methods  were  singled  out  as  needing  additional  study  and  research. 

4.  Approach:  Concurrent  engineering,  also  referred  to  in  other  terms  such  as  "the 
product  development  team  concept,"  is  considered  by  commercial  companies 
as  essential  to  being  competitive.  Concurrent  engineering  reflects  a  true 
system  approach  that  must  be  implemented  from  the  beginning  of  the  product 
development  process.  It  provides  the  means  to  integrate  the  various  and 
sometimes  competing  aspects  of  a  product,  such  as  cost,  producibility, 
reliability,  and  other  performance  requirements.  In  the  specific  case  of 
reliability,  analyses  done  "off-line"  by  a  traditional,  separate  reliability  office 
typically  are  completed  too  late  to  affect  the  design  and  are,  therefore,  useless. 
Businesses  generally  recognize  that  market  success  and  profitability  both 
depend  on  ever-shorter  product  development  cycle  times.  Consequently,  a 
development  approach  with  no  concurrency  is  increasingly  non-competitive  in 
today’s  business  environment. 

Application;  Product  teams,  which  have  authority  and  a  great  deal  of 
autonomy  (sometimes  called  empowerment)  promote  a  feeling  of  "ownership" 
by  all  concerned  in  the  analyses,  assumptions,  and  compromises  made  during 
the  development  process.  Design,  reliability,  quality,  producibility,  risk, 
safety,  and  other  issues  are  all  considered  concurrently  to  meet  aggressive 
development  cycle  times.  A  member  of  the  team  has  the  primary  task  of 
designing  for  reliability,  but  reliability  is  a  team  objective  and  the 
responsibility  rests  with  the  team  leader.  This  member  may  be  called  a 
reliability  engineer,  product  assurance  engineer,  or  design  engineer.  Ideally, 
R&M  analyses,  and  the  necessary  tools,  are  made  a  part  of  the  CAD/CAM 
system  so  that  the  “reliability  engineer”  interacts  with  the  development  team 
in  real-time.  Unfortunately,  those  interviewed  acknowledged  that  many  of  the 
reliability  analyses  are  still  done  “off-line.” 

5.  Approach:  Designing  for  reliability  must  include  the  reliability  of  the 
processes  as  well  as  the  product.  Although  the  subject  of  process  reliability 
(including  the  reliability  of  the  tools  and  manufacturing  equipment)  is  new  to 
the  companies  interviewed,  all  recognize  its  importance.  Those  interviewed 
stated  that  the  traditional  partition  between  design  and  manufacturing  is 
detrimental  to  product  performance  and  quality. 

Application:  Concurrent  engineering  (see  specific  finding  4)  is  helping  to  raze 
the  design-manufacturing  partition  and  helping  both  design  and  manufacturing 
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engineers  better  understand  the  impact  product  design  and  manufacturing  have 
on  each  other  and  the  product.  Design  of  experiments  (DOE)  is  one  technique 
specifically  mentioned  as  an  excellent  tool  for  identifying  the  key  factors 
influencing  manufacturing  quality  and  reliability,  as  well  as  the  reliability  and 
performance  of  the  product. 

6.  Approach:  All  reliability  tasks,  even  those  cited  (in  the  survey)  as  being 
“very  important”,  are  tailored  for  or  deleted  from  a  specific  product 
development  program  based  on  program  needs.  A  cost-benefit  judgment  is 
made  before  a  task  or  other  requirement  is  invoked.  Each  invoked  reliability 
task  is  tailored  so  that  it  best  mitigates  a  specific  risk(s).  Conversely,  if  a 
known  risk  is  not  mitigated,  the  reliability  task  has  no  value. 

Application:  Commercial  companies  emphasize  design  objectives  (such  as  no 
single-point  failures)  and  tailor  the  use  of  only  those  tasks  best  suited  to 
achieving  that  objective.  Commercial  customers  seldom,  if  ever,  impose 
specific  reliability  tasks.  No  specific  means  of  evaluating  the  benefits  (or 
risks)  of  a  given  task  were  evident  from  our  survey  or  interviews.  It  is  a 
matter  of  engineering  judgment  and  experience.  The  decision  to  use  or  delete 
tasks  is  made  by  the  project  manager.  The  manager’s  authority  is  balanced  by 
accountability  for  development,  the  reliability  of  the  resulting  product,  and,  in 
many  instances,  warranty  costs. 

7.  Approach:  Even  for  tasks  included  in  documents  such  as  MIL-STD-785, 
Reliability  Program  for  Systems  and  Equipment  Development  and  Production, 
very  little,  if  any,  administration,  documentation,  or  burden  of  proof  usually 
associated  with  the  standards  is  allowed. 

Application:  Customers  are  neither  given  nor  expect  extensive  documentation 
(other  than  operating  and  maintenance  manuals).  Technical  information,  such 
as  the  content  of  MIL-STDs  and  HDBKs,  is  used  for  guidance  and  “lessons 
learned”  but  is  usually  not  imposed  on  design  teams  by  company  policy  and 
almost  never  by  the  customer.  This  frees  the  commercial  contractor  from  the 
"burden  of  proof  or  administrative  burdens  associated  with  these  documents. 

8.  Approach:  Reliability  tasks  must  not  become  ends  in  and  of  themselves;  the 
focus  must  be  on  the  end  product  and  the  needs  of  the  customer. 

Application:  The  contractor  takes  the  initiative  to  ask  customers  to 
participate  in  one  or  more  design  reviews.  Such  participation  is  intended  to 
ensure  that  the  voice  of  the  customer  has  been  heard  and  properly  understood. 
Contractors  focus  on  the  end  objective,  the  "user  needs,"  and  select  those 
tasks  that  most  effectively  contribute  to  those  objectives.  Tasks  are  tailored 
and  applied  (or  not  applied)  in  accordance  with  the  needs  of  the  specific 
product. 
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9.  Approach:  Parts  application  is  critical.  The  objective  is  to  select  the  part 
most  appropriate  for  the  design  and  stress  levels. 

Application:  None  of  the  companies  interviewed  require  the  use  of  parts 
having  a  certain  quality  level  (i.e.,  a  MIL-SPEC  part).  As  was  stated  by 
several  individuals,  a  MIL-SPEC  part  misapplied  is  a  part  that  will  fail.  That 
is  not  to  say  that  MIL-SPEC  parts  are  not  used  by  commercial  industry.  It 
means  that  the  emphasis  is  on  the  design  engineer  understanding  the  design  and 
stress  levels  and  then  selecting  appropriate  parts.  Robust  designs,  in  which 
relatively  large  shifts  in  a  part’s  characteristics  do  not  affect  system 
performance  and  stress  levels  are  kept  low,  make  it  possible  to  select  parts 
that  might  otherwise  be  excluded  from  consideration. 

10.  Approach:  Strong  supplier  relationships  are  essential.  In  general,  the 
objective  is  to  establish  long-term  relationships,  referred  to  by  most  as 
partnerships,  characterized  by  common  objectives  and  trust. 

Application:  All  the  companies  interviewed  have  a  program  for  reducing  the 
number  of  suppliers  and  for  certifying  those  suppliers.  In  most  cases, 
certification  allows  incoming  inspections  of  supplier-provided  components  to 
be  eliminated  or  reduced  (e.g.,  from  100%  to  periodic  sampling).  Those 
interviewed  also  indicated  that  the  same  requirements  imposed  within  their 
companies  are  imposed  on  suppliers.  Economic  penalties  for  delivering  poor 
product  are  sufficient  to  keep  everyone  honest.  Suppliers  are  not  just  judged 
on  the  unit  costs  of  piece  parts  but  also  on  the  overall  life  cycle  cost  (LCC). 
Included  in  this  LCC  are  purchase  price;  cost  of  inspection,  rejected  parts, 
field  failures,  and  non-conforming  parts;  and  customer  dissatisfaction.  Most 
companies  have  reduced  or  are  reducing  the  number  of  their  suppliers  but  try 
to  keep  two  for  each  critical  item. 
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4.0  CONCLUSIONS 


4.1  THE  PRACTICES 

4.1.1  Reliability  Tasks 

Based  on  the  survey  and  interview  data,  the  tasks  (or  groups  of  tasks)  considered  to  be 
the  most  effective  in  designing  for  reliability  and  offering  the  most  value-added  are: 

Predictions,  simulation,  and  modeling 
Design  reviews 
Subcontractor/vendor  control 

Failure  reporting  and  corrective  action  system  (includes  failure  analysis) 
Development  testing,  such  as  test,  analyze,  and  fix  (TAAF) 

FMEA  and  other  related  analyses 
Design  of  Experiments  (DOE) 

Parts  control 

DOE  is  a  task  identified  through  the  survey  and  interviews  that  is  not  normally  associated 
with  reliability.  It  is  an  excellent  tool  for  identifying  the  key  factors  influencing 
reliability,  performance,  or  manufacturing  quality. 

Commercial  industry  emphasizes  the  objective  of  doing  a  reliability  task,  such  as 
determine  feasibility,  and  uses  all  applicable  tools  to  achieve  that  objective. 

Although  the  survey  and  interview  participants  cited  tasks  that  also  happen  to  be 
included  in  MIL-STD-785,  their  emphasis  is  not  on  performing  a  task,  completing  the 
documentation,  and  complying  with  a  contractual  requirement  but  rather  on  objectives. 
For  that  reason,  it  is  more  important  to  look  at  the  objectives  of  commercial 
reliability  practices  than  at  specific  tasks.  Consider  the  tasks  listed  previously.  As 
shown  in  Table  7,  these  and  other  tasks  can  be  grouped  by  an  R&M  objective  based  on 
the  more  detailed  information  gleaned  from  the  interviews. 

Table  7:  Reliability  Tasks  Grouped  by  Objective 


Objective 

Tasks  that  Contribute  to  Objective 

Determine  feasibility 

Predictions,  simulation,  modeling 

Understand  the  design  and  environment  (eliminate 
single-point  failures,  identify  key  design  parameters, 
understand  failure  mechanisms,  apply  parts  correctly) 

Design  reviews,  FMEA,  FRACAS, 

DOE,  development  testing,  Worst  Case 
Analyses,  Thermal  Analysis,  derating 

Use  proper  parts 

Parts  control,  failure  analysis 

Address  all  sources  of  components,  materials,  etc. 

Vendor/supplier  control 

Validate  the  design 

Development  testing 
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4.1.2  Approach  to  Achieving  Reliability 

Making  reliability  a  design  consideration  from  the  start  of  the  design  process  (i.e.,  an  up¬ 
front  design  approach)  is  essential  to  achieving  a  reliable  product.  Concurrent  engineering, 
product  design  teams,  and  a  systems  approach  are  concepts  that  support  this  belief  in 
up-front  design.  All  are  at  least  part  of  the  vision  of  the  companies  surveyed/interviewed, 
if  not  already  fully  implemented.  Consequently,  reliability  must  be  an  integral  part  of  the 
up-front  design  and  cannot  be  treated  as  a  separate  effort.  Compartmentalizing,  or  stove 
piping,  causes  duplicative  effort,  stifles  information  exchange,  and  is  inefficient. 
Centralizing  some  reliability  functions,  such  as  training  and  the  development  of  tools  and 
policy  is  acceptable.  But  implementation  of  reliability  engineering  must  be  decentralized 
and  must  be  the  responsibility  of  product  design  teams. 

4.2  THE  CONTRACTING  ENVIRONMENT 

As  important  as  the  technical  aspects  (i.e.,  specific  tasks,  the  engineering  approach,  etc.) 
of  commercial  reliability  may  be,  the  contracting  environment  is  equally  if  not  more 
important.  The  contracting  environment  in  the  commercial  sector  is  characterized  in 
Table  8.  Briefly  stated,  the  environment  is  such  that  customers  can  state  their  operational 
needs,  and  the  contractors  can  determine  how  to  meet  that  need.  In  other  words, 
customers  tell  the  contractor  what  is  needed  but  not  how  to  do  it.  The  financial 
rewards  for  success,  competitive  environment,  and  liability  laws  adequately  motivate 
manufacturers  to  keep  the  customer  happy.  These  same  factors  provide  adequate 
remedies  for  the  customer  should  a  manufacturer  fail;  that  is,  the  customer  can  seek 
compensation  through  the  courts  in  extreme  cases  or  simply  buy  from  a  competitor. 

Table  8:  Characterizing  the  Commercial  Sector  Contracting  Environment 


Customer/contractor  relationship 

Trust  and  partnership  with  prompt  and 
direct  remedies  for  failure 

Decision  process 

Few  decision-makers,  minimal  oversight 

Contractor/suppliers  relationship 

Long-term,  partnership 

Economics 

Value,  fitness  for  use  emphasized 

Risk/Payoff 

High  (only  end  results  specified) 

Customer  Requirements 

Functional,  guidelines 

Documentation 

Limited  to  essential  needed  for  design  or 
liability  and  operating  and  support  manuals 

Customer  focus 

Interested  in  product  performance  only 

Contractor  focus 

Primary  focus  on  meeting  customer  needs 
and  being  competitive 

Penalties 

Non-payment,  loss  of  business/markets 

Finally,  the  individuals  who  were  interviewed  are  aware  of  the  efforts  within  the 
Department  of  Defense  to  streamline  acquisition  and  of  the  Congressional  pressures  to 
adopt  commercial  practices.  Based  on  the  comments  of  those  interviewed,  one  must 
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conclude  that  commercial  industry  has  found  that  the  technical  content  of  the  military 
specifications  and  standards  is  very  useful  and  have  based  much  of  their  practice  on  these 
documents.  Neither  the  commercial  nor  the  defense  sector  has  the  definitive  answer  to 
how  to  design  reliable,  maintainable  products.  At  a  technical  level,  the  two  sectors  can 
learn  much  and  profit  from  each  other.  However,  until  and  unless  Federal  laws  and 
regulations  regarding  acquisition  and  procurement  are  changed  to  provide  a  contracting 
environment  that  more  closely  resembles  that  of  the  commercial  sector,  it  is  doubtful  that 
the  defense  industry  will  be  able  to  apply  the  technical  knowledge  in  the  way  commercial 
industry  does. 

4.3  THE  BENCBDMARKS 

Three  major  benchmarks  of  reliability  practice  were  identified  from  the  study  that  reflect 
those  elements  of  a  commercial  approach  to  reliability  most  universally  implemented  or 
deemed  important. 


Completely  analyze  all  failures,  regardless  of  when  or  where  they 
occur  in  development,  to  identify  the  root  cause  of  failure  and  determine 
the  necessary  corrective  action,  including  redesign  and  revision  of 
analytical  tools  and  models. _ 


Avoid  dedicated  reliability  demonstration  testing.  If  required, 
demonstrations  should  focus  on  new  components  or  assemblies,  or  the 
integration  of  old  items  in  a  new  way.  Emphasize  engineering 
development  testing  to  understand  the  and  validate  the  design  process 
and  models.  Accelerated  testing  should  be  used  to  age  high  reliability 
items  and  to  identify  their  failure  mechanisms. _ 


Assign  responsibility  for  reliability  to  an  Integrated  Product  Team  (also 
referred  to  as  a  Product  Development  Team).  Give  the  team  the 
authority  to  determine  the  reliability  requirements  and  to  select  the 
design,  analysis,  test,  and  manufacturing  activities  needed  to  achieve 
that  reliability _ 


4.4  OTHER  KEYS  TO  SUCCESS 

In  addition  to  the  benchmarks  just  listed,  eight  design  and  manufacturing  (Mfg.)  “keys  to 
success”  were  derived  from  the  study  results.  These  keys  are  not  absolutes  (i.e., 
exceptions  to  each  to  accommodate  customer  or  government  requirements  were  evident  in 
the  companies  interviewed)  but  do  represent  a  preferred  way  of  doing  business.  These 
keys  are  described  on  the  following  page. 

DESIGN  Use  an  evolutionary  approach  to  new  product  development.  If  a  new 
KEYS  technology  or  unproved  design  must  be  considered,  invest  additional 

time  and  resources  into  failure  analysis  and  development  testing. 
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MFG. 

KEYS 


Emphasize  design  objectives  (such  as  no  single-point  failures)  and  tailor 
the  use  of  only  those  reliability  tasks  best  suited  to  achieving  that 
objective.  Do  not  allow  reliability  tasks  to  become  ends  in  and  of 
themselves. 

Apply  parts  properly  (rather  than  emphasizing  the  quality  level  of  the 
part). 

Integrate  reliability  analyses  and  modeling  tools  in  computer-based 
design  and  manufacturing  (CAD/CAM)  systems  as  part  of  a  concurrent 
engineering  approach. 

Use  standards  for  guidance  and  “lessons  learned”  but  do  not  impose 
them  on  design  teams.  Do  not  burden  the  use  of  standards  with 
administrative  requirements. 

Use  design  reviews  to  obtain  an  “independent”  peer  assessment  of  the 
design. 

Develop  and  foster  strong,  long-term  relationships  with  a  small 
number  of  suppliers  to  ensure  consistent,  high  quality  of  products, 
keeping  at  least  two  suppliers  for  each  critical  item. 

Recognize  that  reliability  is  impacted  by  the  manufacturing  processes  as 
well  as  the  design  process  and  include  manufacturing  on  the  product 
design  team. 


Reliability  Analysis  Center  (RAC)  •  201  Mill  St.  •  Rome,  NY  13440-6916  •  (315)  337-0900 


Benchmarking  Commercial  Reliability  Practices 


31 


5.0  AREAS  REQUIRING  ADDITIONAL  RESEARCH 


Based  on  the  findings  of  this  study,  the  following  areas  of  additional  research  are 
recommended: 

•  DoD  continue  to  support  research  needed  to  develop  and  update  reliability 
tools,  techniques,  procedures,  etc.  The  results  of  such  research  should 
continue  to  be  made  available  to  industry  either  in  the  form  of  military 
documents  or  as  information  that  can  be  incorporated  in  commercial  standards. 

•  DoD  work  with  industry  and  professional  associations  to  conduct  the  research 
needed  to  develop  reliability  standards  usable  in  both  commercial  and  military 
acquisitions.  All  such  standards,  including  military  standards  on  reliability 
that  are  retained,  should  be  applied  only  as  general  guidelines  in  future 
solicitations  to  encourage  innovative  proposals  by  those  responding. 

Research  should  also  be  conducted  to: 

•  Develop  one  or  more  “model”  R&M  programs  that  manage  buyer 
risks,  ensure  a  reliable  product,  allow  for  appropriate  milestones, 
minimize  use  of  military  standards,  and  are  an  integral  part  of  the  design 
process.  These  model  programs,  which  would  be  coordinated  with 
industry  and  government  experts,  will  be  suitable  for  inclusion  in  RFPs. 

The  models  would  include: 

-  cost-effective  procedural  and  tailoring  guidelines  for  the  key  R&M 
tasks. 

-  guidelines  that  promote  innovative  Drot)osals 

-  performance-oriented  R&M  requirements. 

-  methods  for  evaluating  proposals  (discontinuation  of  a  forced 
common  baseline,  using  a  MIL-STD,  for  example,  will  make 
comparison  of  competing  proposals  potentially  more  difficult  -  this 
will  be  a  major  issue  in  adopting  a  commercial  approach  to 
contracting). 

•  Evaluate  existing  R&M  MIL-STDs,  HDBKs  and  other  documents. 

This  evaluation  should: 

-  identify  alternative  industry  documents  or  develop  alternative 
approaches  that  address  the  risks  and  issues  the  government 
documents  were  intended  to  address. 

-  identify  those  government  documents  having  no  commercial 
counterpart  and  which  should  be  continued;  for  those,  develop 
recommendations  for  revising. 
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-  result  in  efforts  to  develop,  through  professional  and  industry 
associations,  non-government  standards  in  the  areas  of  reliability 
where  none  now  exist. 

•  Determine  the  feasibility  and  value  of  developing  a  means  of 
evaluating  or  measuring  the  effectiveness  of  R&M  tasks  given 
independent  variables  such  as  risk,  level  of  complexity  and  required 
reliability,  criticality  of  product  function,  etc. 

•  Evaluate  specific  reliability  tools  or  types  of  tools  to  identify  those 
having  best  value-added. 

•  Determine  the  level  to  which  reliability  software  tools  have  been 
integrated  into  CAD/CAM  systems  and  the  steps  needed  to  improve 
the  degree  of  integration. 

The  results  of  the  last  three  research  efforts  would  be  used  in  developing  model 
programs,  developing  guidelines  for  evaluating  proposals,  and  so  forth. 
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DEFENSE  ACQUISITION  REFORM 


THE  IMPETUS  FOR  REFORM 

Since  taking  his  position  as  Secretary  of  Defense,  William  J.  Perry  has  taken  several  bold 
actions  to  streamline  and  improve  the  way  in  which  the  Department  of  Defense  (DoD) 
does  business.  These  actions  are  part  of  what  is  called  Defense  Acquisition  Reform 
(DAR).  DAR  is  a  response,  in  part,  to  the  continuing  downsizing  of  the  DoD  and  the 
Military  Services  and,  in  part,  to  recommendations  of  the  Vice  President’s  National 
Performance  Review.  In  a  news  conference  on  June  29,  1994,  the  Secretary  stated: 

More  than  100  years  ago  Victor  Hugo  said  that  more  powerful  than  the 
tread  of  mighty  armies  is  an  idea  whose  time  is  come,  and  integrating  the 
defense  industrial  base  into  the  national  industrial  base  is  an  idea  whose 
time  has  come.  The  Defense  Department  cannot  afford  the  extra  costs 
associated  with  keeping  its  industrial  base  isolated  from  the  national  base, 

...  the  Defense  Department  has  to  basically  and  fundamentally  change 
the  way  it  does  its  procurement.  We  have  to  buy  more  commercial 
products,  we  have  to  make  greater  use  of  commercial  buying  practices,  and 
we  have  to  use  industrial  standards  in  place  of  military  specifications. 

THE  USE  OF  PROCESS  ACTION  TEAMS 

To  develop  his  approach  for  changing  the  DoD’s  business  practices,  Secretary  Perry  is 
using  Process  Action  Team  (PAT),  made  up  of  high-level  government  and  business 
leaders,  and  chartered  at  the  Deputy  Under  Secretary  level.  These  PATs  are  examining 
key  areas,  such  as  procurement  laws  and  practices  and  the  system  of  specifications  and 
standards,  developing  recommendations,  and  submitting  a  final  report  to  the  Secretary  for 
his  review  and  approval. 

SPECIFICATIONS  AND  STANDARDS  PAT 

The  first  such  PAT  was  chartered  by  Colleen  Preston,  Deputy  Under  Secretary  for 
Acquisition  Reform,  and  chaired  by  Darold  Griffin,  former  principal  deputy  of 
acquisition  in  the  Army  Materiel  Command.  The  PAT  was  tasked  to  review  the  system 
of  military  specifications  and  standards.  Specifically,  it  was  told  to  develop 
recommendations  to: 

•  eliminate  unnecessary  and  obsolete  specifications  and  standards 

•  use  performance  specifications  and  standards 

•  use  commercial  standards  and  specifications  to  the  greatest  extent  practicable 

•  encourage  industry  to  propose  alternative  solutions  to  military  specifications 
and  standards 

•  and  reduce  paperwork 
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Specific  recommendations  were  made  by  the  PAT  in  a  final  report^ ^  submitted  to 
Secretary  Perry.  The  Secretary  essentially  accepted  the  recommendations  as  submitted, 
and,  in  a  five-page  memorandum  “Specifications  &  Standards  -  A  New  Way  of  Doing 
Business”  issued  June  29,  1995,  directed  they  be  implemented.  A  five-page  memorandum 
is  rarely  signed  by  a  cabinet-level  Secretary  and  that,  if  for  no  other  reason,  quickly  got 
the  attention  of  the  press  and  of  executives,  managers,  and  technical  personnel  in  the 
military  services  and  military  industries.  Another  indication  of  the  importance  attached 
to  the  memorandum  is  the  fact  that  the  previously  mentioned  news  conference  was  held 
for  the  express  purpose  of  announcing  the  issuance  of  the  memorandum. 

The  intent  of  Perry’s  memorandum  can  be  summarized  as  three  “overarching”  objectives: 

•  Establish  a  performance-oriented  solicitation  process 

•  Implement  a  document  improvement  process 

•  Create  irreversible  cultural  change  in  the  way  DoD  does  business 

As  the  memorandum  states,  military  specifications  and  standards  may  not  be  used  for 
purchasing  new  systems,  major  modifications,  upgrades  to  current  systems,  and  non- 
developmental  and  commercial  items.  Instead,  the  purchasing  agency  must  use 
performance  specifications.  If  it  is  not  practicable  to  use  a  performance  specification, 
then  a  non-government  standard  shall  be  used.  If  it  necessary  to  define  an  exact  design 
solution  because  no  acceptable  non-government  standard  exists  or  because  the  use  of  a 
performance  specification  or  non-government  standard  is  not  cost  effective,  the  use  of 
military  specifications  and  standards  is  authorized  as  a  last  resort  with  an  approved 
waiver.  (Note:  contractors  are  free  to  propose  the  use  of  any  military  document  without 
the  need  for  a  waiver.)  Clearly,  the  intent  is  to  avoid  using  military  specifications  and 
standards,  at  least  those  that  are  not  performance-based,  and  use  commercial 
specifications  and  standards  . 

PERFORMANCE-BASED  REQUIREMENTS 

The  June  29,  1994  memorandum  requires  that  requirements  be  performance-based.  That 
is,  the  requirements  should  describe  how  a  product  should  function,  the  environment  in 
which  it  must  operate,  and  interface  and  interchangeability  characteristics;  they  should  not 
specify  how  the  product  should  be  designed  or  manufactured  -  that  should  be  left  to  the 
supplier.  Table  1  compares  performance-  and  non-performance-based  requirements. 
Four  types  of  performance-based  specifications  have  been  identified  by  the  Defense 
Standards  Improvement  Council.  These  types  are: 

•  commercial  item  descriptions  -  describe  requirements  in  terms  of  function, 
performance,  and  essential  form  and  fit  requirements 


^^Blueprint for  Change,  April  1994.  Changes  were  recommended  in  six  categories:  Policy  Issuance  and 
Changes,  Document  Improvement,  Training,  Automation,  Pollution  Prevention,  and  Miscellaneous 
Acquisition  Reform  Initiatives.  An  implementation  plan  based  on  the  report  was  issued  June  23,  1994. 
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•  guide  specifications  -  standardize  functional  or  performance  requirements  that 
are  common  for  systems,  subsystems,  equipments,  and  assemblies 

•  standard  performance  specifications 

•  program-peculiar  performance  specifications 


Table  1:  Comparing  Performance-Based 
and  Non-Performance-Based  Requirements 


Area  of 
Comparison 

Performance-Based 

Requirement 

Non-Performance-Based 

Requirement 

Purpose 

Describes  functions  product  is  to 
perform 

Describes  how  product  is  to  be 
designed  and  manufactured 

Key  criteria 

Describe  means  for  verifying 
performance 

Describe  means  of  ensuring  specified 
processes  followed 

Design  latitude  given 
to  contractor 

Allows  the  contractor  to  determine 
best  way  to  achieve  results 

Forces  the  contractor  to  use 
prescribed  methods  and  approaches 

Responsibility 

Responsibility  for  results  clearly 
belongs  to  the  contractor 

Responsibility  for  results  shared  by 
customer  and  supplier 

WAIVERS  AND  TIERING 

Two  important  prohibitions  regarding  the  use  of  specifications  and  standards  are  included 
in  the  new  policy  promulgated  by  the  June  29,  1994  memorandum.  First,  a  military 
specification  or  standard  may  be  cited  for  reference  only  unless  a  waiver  is  granted  by  the 
appropriate  Milestone  Decision  Authority.  Second,  a  specification  or  standard  may  not 
reference  other  documents  beyond  the  first  tier.  In  other  words,  if  a  specification 
references  a  second  specification,  any  references  in  the  second  specification  are  ignored. 

IMPACT  OF  THE  SPECIFICATIONS  AND  STANDARDS  MEMORANDUM 

As  a  result  of  the  memorandum,  the  preparing  activities  of  military  specifications  and 
standards  are  reviewing  their  documents  and  will  be  recommending  disposition  to  the 
Secretary.  One  of  eight  possible  recommendations  for  disposition  of  a  military 
specification  or  military  standard  may  be  made.  They  are: 

•  Retain  as  performance-based  document  (some  revision  may  be  necessary) 

•  Retain  as  interface  standard 

•  Retain  as  test  method  standard 

•  Convert  to  handbook 

•  Inactivate  for  new  design  (reprocurement  only) 

•  Delete  in  favor  of  a  commercial  item  description 

•  Delete  in  favor  of  a  non-government  standard 

•  Cancel 
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From  articles  on  the  subject  printed  in  The  Defense  News,  the  DoD  Standardization 
Newsletter,  and  other  publications,  most  observers  anticipate  that  the  majority  of 
documents  will  be  deleted,  either  outright  or  in  favor  of  a  non-government  standard(s). 
This  anticipation  is  strongest  regarding  those  standards  dealing  with  functional  areas,  such 
as  reliability  and  maintainability,  systems  engineering,  and  configuration. 

Implementation  of  Secretary  Perry’s  direction  by  the  military  services  has  been  uneven 
and  inconsistent.  In  some  cases,  procurement  activities  have  forbade  the  inclusion  of 
military  specifications  and  standards,  even  when  used  only  for  guidance.  Inclusion  of 
these  documents  for  guidance  only  is  permitted  without  the  need  for  a  waiver^^.  in  other 
cases,  contractors  have  been  alerted  by  procuring  activities  to  not  include  military 
specifications  and  standards  in  their  proposals.  This  latter  action  is  in  conflict  with  the 
intent  of  the  Perry  memorandum.  Clearly,  much  work  remains  to  be  done  regarding 
implementation. 

ADOPTING  COMMERCIAL  PRACTICES 

Certainly  the  Department  of  Defense  and  the  defense  industry  can  learn  from  the  changes 
commercial  industry  has  made  to  remain  competitive  in  the  face  of  foreign  competition.  It 
is  also  true  that  the  technical  content  of  the  military  specifications  and  standards  reflects 
years  of  “lessons  learned”  that  have  proved  useful  to  commercial  industry.  Much  of 
commercial  practice  is  based  on  these  documents.  It  is  probably  safe  to  say  that  neither 
the  commercial  nor  the  defense  sector  has  the  definitive  answer  to  how  to  do 
procurement.  It  is  also  true  that  the  contracting  environment,  its  rules,  motivations,  and 
laws,  in  the  commercial  sector  is  much  different  than  that  of  the  government. 

In  the  commercial  sector,  procurement  serves  the  technical  needs  of  the  company. 
Purchases  are  made  in  a  way  that  best  satisfies  the  technical  need  at  an  affordable  price. 
Products  may  be  bought  without  any  legal  concern  for  “competing”  the  procurement. 
Once  a  company  finds  a  “good”  source  for  an  item,  it  can  continue  to  do  business  with 
that  source  until  it  finds  a  better  source.  Profit  levels  are  limited  only  by  the  law  of 
supply  and  demand  and  the  company’s  ability  to  operate  efficiently. 

In  the  government  sector,  concerns  for  small  businesses  and  minority-owned  businesses, 
the  desire  to  have  free  and  open  competition,  acknowledgment  of  the  government’s  large 
buying  power,  and  other  issues  complicate  procurement.  Technical  concerns  must  often 
take  a  back  seat  to  procurement  legal  issues.  Suppliers  must  periodically  recompete 
regardless  of  well  they  have  been  doing  or  how  satisfied  the  technical  customer  may  be 
with  the  product.  Profits  are  limited  by  law,  often  at  a  much  lower  level  than  that  of  the 
commercial  sector. 


^^Confirmed  by  government  representatives  to  a  Society  of  Logistics  Engineers  (SOLE)  sponsored 
conference,  Specifications  and  Standards  in  Transition,  Crystal  City,  VA,  January  26-27,  1995,  and  by 
articles  in  the  DoD  Standardization  Newsletter. 
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A  FINAL  THOUGHT 


Defense  Acquisition  Reform  is  essential  just  as  continual  improvement  is  essential  to  a 
commercial  company.  It  is  important,  especially  as  defense  budgets  continue  to  decrease, 
to  find  more  efficient  and  effective  processes  to  buy  the  products  needed  to  support  the 
military  services.  Examining  the  way  in  which  commercial  industry  does  business  is  a 
valid  and  worthwhile  step  in  improving  defense  acquisition.  In  many  respects,  the 
Department  of  Defense  is  benchmarking  certain  processes  using  commercial  industry  as 
the  exemplar.  When  a  process  used  by  commercial  industry  is  superior  to  that  used  by 
the  Department  of  Defense  or  its  contractors,  then  that  process  should  be  adopted.  In 
some  cases,  however,  commercial  processes  may  not  be  usable,  given  Federal 
procurement  laws;  in  some  instances,  the  defense  industry’s  process  may  actually  be 
better. 

The  defense  and  commercial  sectors  can  learn  much  and  profit  from  each  other  and  should 
take  the  necessary  actions  to  adopt  the  other’s  practices  when  it  promises  improvement. 
However,  Federal  laws  and  regulations  regarding  acquisition  and  procurement  are  an 
obstacle  to  the  government  adopting  many  commercial  practices.  Until  and  unless  these 
laws  and  regulations  are  changed  to  provide  a  contracting  environment  that  more  closely 
resembles  that  of  the  commercial  sector,  it  is  doubtful  that  the  Department  of  Defense 
and  the  defense  industry  will  be  able  to  fully  adopt  and  apply  commercial  practices. 
Hopefully,  the  PAR  chartered  to  examine  procurement  laws  and  regulations  will 
recommend  ways  to  eliminate  the  obstacle. 
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ABSTRACT 

THE  INTENT  OF  THIS  DOCUMENT  IS  GIVE  COMPANIES  INTERESTED  IN  IMPLEMENTING  CONCURRENT  ENGINEERING  SPECIFIC  INSIGHT  INTO 
HOW  TO  GET  STARTED.  IT  IS  INTENDED  TO  BE  A  LIVING  DOCUMENT  AND  TO  BE  PERIODICALLY  UPDATED  AS  MORE  INFORMATION 
BECOMES  AVAILABLE. 
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Source  Title/Author  Summary 

RAC  17297-  Commercial  vs.  DoD  Typical  Military  Reliability  Program: 

051  Reliability  Programs 
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APPENDIX  C 


SURVEY  FORMS 


I,  SURVEY  INTRODUCTION 


08/23/93 


This  survey  is  being  conducted  by  the  Reliability  Analysis  Center  (RAC),  an  Information 
Analysis  Center  (lAC)  chartered  by  the  Department  of  Defense  (DoD)  to  serve  as  a  government 
and  industry  focal  point  to  improve  the  reliability,  maintainability,  and  quality  of  manufactured 
components  and  systems. 

DoD  is  concerned  that  its  historic  reliability  and  maintainability  (R&M)  standards, 
handbooks,  and  other  documents  do  not  adequately  address  the  advanced  R&M  practices  used  by 
commercial  industry.  Thus,  the  RAC  has  been  tasked  to  baseline  how  commercial  industry, 
spurred  by  an  increasingly  competitive  marketplace,  is  designing  and  producing  systems  to  be 
reliable  and  easily  and  economically  maintained. 

DEFINITIONS 

Terms  used  in  this  survey  may  not  be  universally  defined  among  commercial  and  defense 
industries.  For  the  purpose  of  this  survey,  the  following  definitions  apply: 

1 .  Reliability.  The  probability  that  a  item  will  perform  satisfactorily  for  at  least  a  given  time 
interval  when  used  under  specified  conditions.  May  also  be  defined  as;  mission  success 
probability  (Pg),  mean  time  between  failure  (MTBF),  the  time  to  failure,  the  rate  of  failure  or 
failures  per  operating  interval. 

2.  Maintainability.  The  probability  that  a  failed  item  will  be  restored  to  operable  condition  in  a 
specified  time.  It  may  also  defined  as;  the  time  to  restore  a  system  mean  time  to  repair 
(MTTR),  maximum  corrective  down  time,  or  the  resources  required  to  operate  the  system 
over  a  given  time  interval. 

3.  Item  life  cycle  Phases: 

a.  Phase  1,  Development  Phase  during  which  a  concept  is  "translated"  into  a  detailed 
design.  Activities  include  design,  trade  studies,  configuration  management,  development 
of  technical  data  package,  and  development  testing. 

b.  Phase  2,  Production  Phase  during  which  the  item  is  produced  or  manufactured. 

Activities  include  parts  control,  screening,  process  control,  tooling,  testing,  and 
fabrication. 

c.  Phase  3,  Operational  Phase  during  which  the  system  is  used  and/or  maintained  by  the  end 
use  customer. 
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4.  Specific  R&M  tasks. 

a.  FMECA.  Failure  Modes,  Effects,  and  Criticality  Analysis. 

b.  FTA.  Fault  Tree  Analysis 

c.  TA.  Thermal  Analysis. 

d.  SCA.  Sneak  Circuit  Analysis. 

e.  WCA.  Worst  Case  Analysis. 

f  PRED.  Prediction  of  Reliability  &  Maintainability  performance. 

g.  ALLOC.  Allocation  of  Reliability  &  Maintainability  requirements  to  lower  level 
equipments. 

h.  MODEL.  Reliability  modeling  (block  diagrams,  math  models,  etc.) 

i.  CENVTRON.  Characterization  of  Environment  (operating,  handling,  transportation  & 
storage) 

j.  DRs.  Design  reviews. 

k.  PC.  Parts  Control. 

l.  S/VC.  Subcontractor  /  Vendor  Control 

m.  ESS.  Environmental  Stress  Screening. 

n.  TAAF.  Test,  Analyze,  and  Fix. 

1.  RGT.  Reliability  Growth  Testing. 

p.  RQT.  Reliability  Qualification  Testing. 

q.  FRACAS.  Failure  Reporting  Analysis,  and  Corrective  Action  System. 

r.  MDEMO.  Maintainability  Demonstration. 
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II.  SURVEY  QUESTIONNAIRE 
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Given  a  choice,  please  indicate  your  selection  by  circling  the  appropriate  response. 

1.  COMPANY  NAME 


2.  COMPANY  ADDRESS 


3.  COMPANY  STANDARD  INDUSTRIAL  CODE  (SIC)  &  MAJOR  PRODUCT(S) 


4.  PRIMARY  PRODUCT  INFORMATION 

a.  Production  volume  (units  per  year);  _ _ 

b.  Product  Criticality;  High  (human  life  affected) _ 

Moderate  (property  damage  could  result) _ 

Low  (safety  not  normally  an  issue)  _ 

c.  Approximate  Unit  cost;  _ 

e.  What  governmental  regulations  apply  regarding  the  sale,  use,  or  safety  of  this  product? 


f  Are  there  applicable  industry  association  (e  g.  SAE,  EEEE,  EPRI,  NEMA)  guidance/policy 
documents.  Y  N 

If  Yes  to  either  e  or  f,  please  explain. _ 


5.  PERSON  COMPLETING  THIS  SURVEY. 

a.  Name _ 

b.  Title _ 

c.  Telephone  No. _ FAX  No. 

e.  Mail  Stop  (if  applicable) _ 


1 
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f.  Are  you  responsible  for  any  or  all  aspects  of  product  reliability,  Y  N 
and/or  maintainability?  Y  N 

6.  R&M  PROGRAM  MANAGEMENT 

a.  Is  there  a  dedicated  R&M  organization(s)  within  your  company?  Y  N 

b.  If  you  answered  no  to  the  last  question,  are  R«S:M  explicitly  addressed  by; 

Design  Engineers  Production  Engineers  Both 

c.  Is  R&M  training  or  minimum  education/experience  required  of  those  responsible  for 

product  R&M?  Y  N 

d.  Are  vendors  and  suppliers  required  to  have  a  defined  R&M  program?  Y  N 

7.  R  &  M  TASK  EFFECTIVENESS 

a.  Do  you  measure  the  effectiveness,  importance  or  value-added  of  the  R&M  tasks  which 

you  perform?  Y  N 

b.  If  you  answered  Yes,  please  explain  how  you  measure  R&M  task  effectiveness. 


8.  DoD  Recognized  R&M  TASKS 

For  each  of  the  tasks  on  the  following  page,  indicate  whether  your  company  performs  the  task,  its 
relative  importance,  whether  the  task  is  required  by  the  customer  (C)  or  company  policy  (P),  and 
the  phase(s)  in  which  the  task  is  performed. 
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TASK  PERFORMED?  IMPORTANCE  REQUIRED  PHASE 
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9.  ADDITIONAL  TASKS 

Section  8  listed  R&M  tasks  frequently  required  by  military  specifications  and  standards.  Please 
list  and  describe  any  other  tasks  your  company  requires  and/or  generally  performs  to  ensure 
product  R&M.  Use  additional  sheets  if  needed.  For  each  task,  indicate  its  effectiveness  (on  a 
scale  of  I  to  3,  as  before)  and  the  phase(s)  in  which  it  is  performed  (again,  circle  the  appropiate 
response). 

Task  1 :  Description _ 


Importance: 

1 

2 

3 

Phases: 

Development 

Production 

Operation 

Task  2: 

Description 

Importance; 

1 

2 

3 

Phases; 

Development 

Production 

Operation 

Task  3: 

Description 

Importance: 

1 

2 

3 

Phases: 

Development 

Production 

Operation 

Task  4: 

Description 

Importance. 

I 

2 

3 

Phases. 

Development 

Production 

Operation 

10.  TASK  IMPLEMENTATION 


Do  you  use  any  international,  industrial  or  US  government  standards  (STDs),  handbooks 
(HDBKs),  or  models  to  implement  any  of  the  tasks  from  sections  8  and  9?  Y  N 

If  yes,  please  list  the  task  number  and  corresponding  applicable  references. 

TASK  No.  INTERNATIONAL  /  INDUSTRIAL  /  GOVERNMENT 

STD /HDBK/ MODEL 


Thank  you  for  your  help  in  this  task  by  completing  this  survey  form. 
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SERVICE  COMPANIES 


III.  SURVEY  QUESTIONNAIRE 

Given  a  choice,  please  indicate  your  selection  by  circling  the  appropriate  response. 

1.  COMPANY  NAME 

2.  COMPANY  ADDRESS 


3.  COMPANY  STANDARD  INDUSTRIAL  CLASSIFICATION  CODE  &  MAJOR  PRODUCT(S) 


4.  PERSON  COMPLETING  THIS  SURVEY. 

a.  Name  _ _ 

b.  Title _ 

c.  Telephone  No. _ 

d.  FAX  No. _ 

e.  Mail  Stop  (if  applicable) _ 

f.  Are  you  responsible  for  any  or  all  aspects  of  product  /service  reliability?  Y  N  ; 
maintainability?  Y  N 

5.  R&M  OF  PRODUCT 

a.  Is  there  an  organization  within  your  company  responsible  for  product/service  R&M? 

Y  N 

b.  Do  you  communicate  to  your  suppliers  required  levels  of  R&M  for  your  equipment 
or  specific  R&M  tasks  that  you  want  them  to  perform?  Y  N  If  you  answered  yes,  please 
explain  how  you  communicate  these  requirements  (through  specifications,  interchange 
meetings,  standards,  etc.). 
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6.  DoD  Recognized  R&M  TASKS 


If  you  do  require  that  suppliers  perform  specific  R&M  tasks,  please  indicate  which  of  the 
following  are  required,  its  relative  importance,  and  why  you  require  it 


TASK  REQUIRED? 


FMECA  Y 

FTA  Y 

TA  Y 

SCA  Y 

WCA  Y 

Parts  Control  Y 

Derating  Y 

Predictions  Y 

Allocation  Y 

Modeling  Y 

DRs  Y 

FRACAS  Y 

ESS  Y 

TAAF  Y 

RGT  Y 

RQT  Y 

MDEMO  Y 

S/VC  Y 


N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 


IMPORTANCE 
Least  Most 
1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 


7.  ADDITIONAL  TASKS 

Please  list  and  describe  any  other  tasks  your  company  requires  to  address  product  R&M. 

Use  additional  sheets  of  paper  if  needed.  For  each  task,  indicate  its  performance  (on  a  scale 
of  1  to  3  as  before). 

Task  1 :  Description _ 

Importance  1  2  3 

Task  2:  Description _ 

Importance  1  2  3 

Task  3;  Description _ 

Importance  1  2  3 

Task  4:  Description _ 

Importance  1  2  3 

8.  TASK  EFFECTIVENESS 

a.  Do  you  measure  the  effectiveness  or  value-added  of  the  R&M  tasks  or  levels 
you  require?  Y  N 

b.  If  you  answered  Yes  to  8a,  please  explain  how  you  measure  effectiveness  or 
value  added. 
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9.  TASK  IMPLEMENTATION 

Do  you  use  any  international,  industrial,  or  US  government  standards  (STDs),  handbooks 
(HDBKs),  or  models  to  determine  the  tasks  or  R&M  levels  you  require?  Y  N 

If  yes,  please  list  the  task  and  corresponding  applicable  references. 

TASK  INTERNATIONAL/INDUSTR1AL7GOVERNMENT 

STD/HDBK/MODEL 


Thank  you  your  help  in  this  task  by  completing  this  survey  form. 
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APPENDIX  D 

INTERVIEW  QUESTIONS  AND  NOTES 
Section  1:  Interview  Questions 
Section  2:  Notes 


APPENDIX  D 

SECTION  1 
Interview  Questions 


INTERVIEW  QUESTIONS 
FOR  ON-SITE  VISITS  IN  SUPPORT  OF 
BENCHMARKING  OF  COMMERCIAL  R&M  PRACTICES 


Section  1  .-Customer  Relationships 

1.  Broadly  identify  your  product  line.  Define  "failure"  in  the  context  of  your  equipment.  What 
are  the  consequences  of  "failure"? 

2.  Do  you  sell  to  the  end  user  or  to  an  intermediary  (OEM,  etc.)? 

3.  Are  your  products  standard  catalog  items,  or  is  customization  involved?  To  what  level  of 
detail  do  customers  specify  the  product  in  their  orders? 

4.  Are  you  responsible  for  maintenance  or  repair  of  products  in  any  way?  Is  there  an  express 
or  implied  warranty?  If  so,  who  does  warranty  repairs?  How  are  warranty  costs  tracked?  Are 
repair  costs  factored  into  product  pricing?  Are  designers  taught  to  consider  maintenance  in 
their  designs? 

5.  Discuss  the  typical  customer  relationship.  Is  your  company  pre-qualified  by  audit  or  other 
means?  Does  the  customer  inspect  or  audit  production  or  design?  What  documentation  is 
required  by  the  customer  along  with  the  product?  How  is  your  delivered  product  inspected  by 
the  customer  prior  to  acceptance?  How  does  quality  or  reliability  affect  your  status  as  a 
supplier? 

6.  Discuss  what  happens  when  a  customer  is  unhappy  with  your  product.  How  is  the  problem 
identified  by  the  customer?  How  (who)  do  they  notify  your  company  ?  What  is  expected  of 
your  company  to  correct  the  problem? 

7.  How  much  feedback  do  you  get  regarding  field  performance  of  your  product?  Do  you  have  a 
field  performance  database?  Do  you  perform  failure  analysis  on  failed  units?  How  is  this 
knowledge  fed  back  to  design/manufacturing? 
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INTERVIEW  QUESTIONS 
FOR  ON-SITE  VISITS  IN  SUPPORT  OF 
BENCHMARKING  OF  COMMERCIAL  R&M  PRACTICES 

Section  2. -Product  Engineering  Practices 

1 .  In  your  company,  who  is  ultimately  responsible  for  the  quality  and  reliability  of  product? 
How  does  this  person  manage  to  achieve  quality  and  reliability?  What  measures  are  used  to 
monitor  performance?  How  are  these  features  monitored  during  Design?  Production? 

2.  How  does  your  company  design  its  products,  and  the  processes  used  to  manufacture  those 
products,  to  reduce  the  chance  or  consequences  of  failure  during  operation?  What  do  you 
name  this  aspect  of  the  system  performance? 

3.  To  what  extent  is  your  product  design  subject  to  government  regulations?  OSHA?  FCC? 

UL?  _ _ (other)  Are  you  required,  by  any  governmental  regulations  or  laws,  to 

address  safety,  quality  or  reliability  of  system  performance? 

4.  Is  this  aspect  of  system  performance  specified  by  the  customer? 

a.  If  so,  how  and  in  what  terms  is  it  specified?  Monitored?  Controlled? 

b.  If  not,  what  are  the  incentives  for  addressing  this  aspect  of  system  performance? 

5.  How  do  you  communicate  reliability  or  quality  aspects  of  system  performance  to  your 
design  staff?  Your  manufacturing  engineers?  Your  managers?  Your  customers? 

6.  What  tasks  do  you  perform  to  ensure  the  achievement  of  this  aspect  of  system  performance? 

a.  Please  explain  each  task: 

1)  What  does  the  task  entail? 

2)  When  is  it  done  in  the  product  development  cycle? 

3)  Who  performs  it? 

b.  By  what  criteria  are  specific  tasks  selected? 

c.  Who  decides  that  a  specific  task  be  performed? 

d.  How  do  you  measure  the  effectiveness  or  value-added  of  a  task  or  a  group  of  tasks? 

d.  What  documentation  is  generated  in  conjunction  with  these  tasks? 

e.  For  what  purpose  is  this  documentation  generated? 
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INTERVIEW  QUESTIONS 
FOR  ON-SITE  VISITS  IN  SUPPORT  OF 
BENCHMARKING  OF  COMMERCIAL  R&M  PRACTICES 

7.  What  area  of  area  of  management  do  these  tasks  fall  within?  Who  controls  the  budget  for 
these  activities? 

8.  Describe  the  integration  of  these  task  with  each  other  and  with  other  areas  of  management 
(e.g..  Design,  Quality,  Manufacturing,  etc.) 

9.  Describe  any  system  you  use  to  obtain  failure  information  feedback  from  your  customers. 

10.  If  any  of  the  tasks  you  now  perform  are  required  by  regulation,  law,  or  the  customer,  which, 
if  any,  would  you  not  perform  if  you  had  a  choice  (i.e.,  you  see  no  value-added  but  perform 
them  only  because  they  are  required)? 

11.  Do  you  specify  this  aspect  of  system  performance  to  your  suppliers? 

a.  How  do  you  specify  it? 

b.  How  do  you  measure  their  success  in  meeting  your  requirements? 
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INTERVIEW  QUESTIONS 
FOR  ON-SITE  VISITS  IN  SUPPORT  OF 
BENCHMARKING  OF  COMMERCIAL  R&M  PRACTICES 


Section  3. -Supplier  Relationship  and  Controls 

1 .  Explain  how  you  manage  quality  and  reliability  of  products  purchased  from  suppliers.  How 
is  Q/R  specified?  measured?  What  happens  when  the  quality  of  vendor  product  is  unacceptable? 

2.  What  audits  or  other  assessments  are  performed  in  order  to  qualify  a  potential  supplier? 

3.  What  documentation  is  required  with  a  product  to  support  Q/R/Safety  of  the  product? 
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APPENDIX  D 

SECTION  2 
Notes 


SummaiY  of  Interview  of  Company  A  Reoresentativefs).  This  company’s  products  are  sold  to 
distributors  who  sell  to  the  end  customers.  The  company’s  greatest  challenge  is  converting  the  voice  of  the 
customer  into  a  design.  Specifications  are  developed  internally  based  on  customer  surveys,  warranty  data, 
benchmarking  of  competitors'  products,  and  dedicated  service  testing.  Failures  are  defmed  as  any  incident  that 
causes  the  customer  to  bring  the  product  in  for  service.  Reliability  is  a  key  design  requirement.  Warranty  data 
resulting  from  such  "returns”  are  used  in  evaluating  reliability  as  are  the  results  of  customer  surveys.  The  latter 
are  used  to  determine  customer  tolerances  to  a  variety  of  performance  features.  Information  from  the  surveys 
and  warranty  data  is  used  to  develop  specifications.  The  interviewee(s)  believes  that  the  amount  of  data 
available  from  warranty  repairs  is  more  than  sufficient,  but  the  data  are  lacking  in  quality  (i.e.,  improper  codes, 
insufficient  information  on  causes,  etc.). 

The  interviewee(s)  sees  two  facets  of  design:  deterministic  and  probabilistic.  In  converting  the  voice  of 
the  customer,  most  engineers  attempt  to  determine  the  effect  of  every  cause,  and  the  cause  of  every  effect.  This 
approach  is  deterministic  and  the  basis  for  most  engineering  curricula.  Statisticians,  on  the  other  hand,  attempt 
to  use  probability  to  account  for  chance  and  unknowns.  This  representative(s)  believes  that  a  robust  design 
philosophy  is  achievable  only  through  a  "dynamic  tension"  between  statisticians  and  engineers  (i.e.,  between 
deterministic  design  and  probabilistic  design).  The  result  is  probabilistic  design.  Besides  probabilistic  design 
(i.e.,  robust  design),  the  following  tasks  were  specifically  mentioned  as  important  to  the  company’s  design 
approach:  benchmarking,  Failure  Modes  and  Effects  Aiialysis  (FMEA),  design  of  experiments  (DOE),  life 
testing,  development  testing  that  is  similar  to  growth  testing,  computer-aided  design/manufacturing 
(CAD/CAM),  Statistical  Process  Control  (SPC),  and  concurrent  engineering.  Failure  analysis  of  failed  parts 
from  the  field  is  performed  but  is  really  too  late  for  the  fielded  product;the  information  is  best  used  to  prevent 
failures  in  the  next  product.  Competitors'  products  are  tom  down  to  establish  benchmarks. 

Products  are  developed  using  a  four-phase  approach:  Concept  Phase  (0),  Prototype  Phase  (1), 
Development  Phase  (2),  and  Production  Phase  (3).  In  the  Concept  Phase,  normally  1  year,  the  engineers 
perform  an  FMEA  and  try  to  understand  the  physics  of  design.  In  the  Prototype  Phase,  parametric  design  is 
done.  Design  of  Experiments  is  used  in  Phase  2,  Development,  to  fine  tune  the  design  and  validate  the 
FMEA.  Finally,  the  product  enters  Production  Phase  in  which  SPC  is  a  key  tool  used  to  ensure  quality. 
Products  are  developed  by  teams  and  are  under  the  direction  of  an  Engineering  Director  (ED).  The  ED  is 
responsible  for  the  design  and  cost  of  the  design.  The  ED  reports  to  the  Program  Manager  who  has  total 
responsibility  for  the  vehicle,  including  manufacturing  and  warranty  costs. 

The  development  teams  consist  of  representatives  from  engineering,  drafting,  purchasing,  design,  and 
manufacturing.  This  team  approach  is  the  company’s  version  of  concurrent  engineering.  A  sophisticated 
CAD/CAM  system  is  used  to  translate  "art  to  part."  Based  on  interpreting  the  voice  of  the  customer  and  from 
fundamental  physics,  a  disciplined  system  is  used  to  identify  the  control  characteristics  that  must  be  a  part  of 
the  design  (similar  to  Quality  Function  Deployment).  Control  characteristics  are  then  translated  into  design 
parameters.  The  team's  objectives  are  to  achieve  each  control  characteristic  at  the  lowest  cost;  these  objectives 
are  called  Targets  of  Excellence.  The  team  approach  results  in  good  integration  of  safety,  reliability,  and 
quality.  Reliability  and  quality  are  key  considerations  in  eveiy  phase.  Reliability  of  product  and  process  is 
addressed,  although  the  latter  is  relatively  new,  not  fully  developed,  and  secondary  to  the  former. 

Reliability  predictions  are  performed  early  in  Phase  0,  using  MIL-HDBK-217  for  electronics. 
Allocations  are  not  made;  the  emphasis  is  on  design  for  reliability.  To  respond  to  the  voice  of  the  customer, 
the  reliability  requirements  for  each  part  are  set  so  high  that  they  cannot  be  measured  (in  a  practical  and 
economical  sense)  using  statistical  demonstrations.  An  FMEA  is  done  at  the  system  and  component  levels. 
Component-level  FMEAs  essentially  are  done  once,  not  for  each  successive  application  of  a  component.  The 
operating  environment  of  each  application  is  reviewed  to  ensure  that  the  FMEA  is  still  applicable.  Duane’s 
theories  are  considered  too  deterministic,  so  growth  testing,  in  the  traditional  sense,  is  not  performed. 
However,  testing  is  done  during  Phase  0  to  learn  about  the  physics  of  failures  associated  with  the  design. 
Understanding  the  design  as  early  as  possible  is  critical  because  there  is  never  enough  money  during  design  and 
because  the  cost  of  failures  increases  as  the  design  progresses.  Partly  to  create  a  design  audit  trail,  but  for 
primarily  for  liability  reasons,  all  tasks  are  thoroughly  documented. 

Most  of  the  design  criteria  come  from  internal  specifications  and  standards.  However,  the  company’s 
products  are  subject  to  some  government  regulations.  Some  government-witnessed  testing  is  required.  No 
company  "Reliability  and  Maintainability  (R&M)  Standards"  exist;  R&M  can  drive  the  design  only  through 
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the  efforts  of  the  design  team  and  the  direction  of  the  EDs.  For  products  sold  in  Europe,  International 
Standards  Organization  (ISO)  9000  requirements  must  be  met. 

During  production,  SPC  is  the  primary  means  of  controlling  product  quality.  Incoming  inspections  of 
parts  and  components  are  made  or  not  made  depending  on  the  part  or  component  and  the  supplier. 

Success  of  rehability  efforts  is  ’’measured"  using  the  following  criteria: 

1 .  Number  of  repeat  customers 

2.  Age  of  buyer  (i.e.,  reaching  intended  market) 

3.  Number  of  incidents  per  product 

4.  Warranty  costs  and  frequency  (the  latter  is  considered  the  most  important  of  the  two) 

5.  Number  of  maintenance-induced  problems 

The  long-term  goal  of  the  company  is  to  reduce  the  number  of  its  suppliers  and  to  establish  long-term 
relationships  with  the  "best"  suppliers.  Suppliers  are  required  to  meet  the  control  characteristics  developed  by 
the  company  for  the  particular  part  or  component.  The  best  suppliers  will  be  those  who  can  satisfy  the  control 
characteristics  at  the  lowest  cost  (thereby  meeting  the  Targets  of  Excellence).  Incoming  inspections  are  still 
performed  for  selected  parts  and  components,  depending  on  the  supplier.  An  objective  of  the  company  is  to 
eliminate  incoming  inspections  by  working  with  the  suppliers  to  certify  processes  and  by  establishing  a  level  of 
confidence  and  trust. 

Summary  of  Interview  of  Company  B  RepresentativetsV  This  world-wide  manufacturing  organization 
sells  its  products  directly  to  customers  and  to  customers  through  distributorships.  Products  are  somewhat 
standard,  but  some  highly  customized  products  are  produced.  Specifications  are  driven  by  maiketing  research 
(and  some  benchmarking  of  competitors’  products)  and  the  end  user.  No  standard  method  is  used  to  specify 
product  performance  other  than  to  attempt  to  stay  one  step  ahead  of  the  competition.  In  addition,  specifications 
from  Underwriters  Laboratory  (UL)  and  the  government  govern  performance  requirements  of  the  products. 
Specifications  for  equipment  manufactured  in  Europe  are  linked  to  ISO  9000. 

The  product  development  cycle  is  driven  by  market  timing  for  a  certain  product  and  is  the  absolute 
responsibility  of  a  project  manager.  The  company  has  no  policy  on  procedures  for  the  development  phase. 
Project  managers  can  institute  any  method  of  development  that  they  feel  will  guarantee  project  completion 
within  the  development  time  schedule.  Testing  and  reliability  engineering  are  two  phases  of  development  that 
are  most  often  eliminated  to  meet  the  product  development  schedule.  Extensive  service  commitment  is  very 
often  instituted  for  a  new  product  line  to  identify  and  correct  problem  areas  as  they  occur  in  the  field.  Although 
certain  reliability  tasks  were  cited  as  being  important,  they  are  only  sporadically  included  in  product 
development  efforts  (especially  true  for  the  custom  products).  It  is  entirely  up  to  the  product  manager  whether 
to  perform  a  specific  analysis  or  test  (for  reliability  or  any  other  puipose).  Furthermore,  program  managers  are 
primarily  schedule-driven.  Poorly  performing  products  are  fielded  to  meet  the  schedule,  with  the  performance 
problems  addressed  through  service  or  engineering  changes  in  the  field. 

The  con:q)any  rates  the  field  performance  of  its  product  lines  by  the  service  cost  associated  with  each  line. 
The  service  call  rate  or  frequency  of  service  calls  is  of  a  much  lesser  importance.  This  method  of  evaluating 
field  performance  is  focused  on  controlling  warranty  and  service  cost,  not  on  increasing  availability  and 
reliability.  Although  increased  attention  to  reliability  during  design  would  be  an  effective  way  of  controlling 
service  costs,  most  managers  want  to  minimize  development  time.  Frequent,  low-cost  repairs  are  favored  over 
improving  reliability.  Even  though  the  service  cost  may  be  low  with  this  approach,  Ae  availability  of  the 
machine  will  be  low  and  the  frequency  of  failures  an  annoyance  to  the  customer. 

Analysis  of  failed  items  is  rarely  performed  unless  the  complexity  of  the  unit  and  cost  of  the  repair  begin 
driving  up  the  service  cost.  Documentation  of  the  service  calls  does  not  provide  sufficient  data  to  determine  the 
cause  of  failure,  and  the  interviewees  gave  no  indication  that  post  warranty  field  data  are  analyzed  to  any 
significant  degree. 

The  Reliability  Department  is  part  of  a  larger  organization  responsible  for  overall  quality  assurance.  Two 
individuals  are  responsible  for  providing  R&M  training  to  design  groups  throughout  the  company.  These 


D-2-3 


individuals  have  a  through  understanding  of  reliability  engineering  and  have  access  to  a  comprehensive 
reliability  library. 

Many  of  the  Reliability  Department’s  efforts  are  to  promote  its  services.  Staff  members  advertise  and 
explain  the  value  of  their  reliability  services  in  the  training  courses.  These  training  courses  range  from  an 
introduction  to  the  basics  of  reliability  (e.g.,  mathematics,  terminology,  FMEA,  statistics)  to  a  nine-session 
coiuse  on  testing  practices  and  procedures.  It  is  the  intent  of  the  Reliability  Department  to  interest  the  various 
project  teams  in  the  reliability  discipline  and  then  provide  support  by  applying  advanced  reliability  tools  and 
techniques  to  specific  product  development  efforts.  The  decision  to  implement  asty  (or  no)  reliability  task  is 
made  solely  by  each  project  manager,  in  most  cases  no  tasks  are  implemented.  The  imderlying  reason  that 
reliability  tasks  are  not  usually  implemented  is  that  the  project  managers  feel  that  reliability  engineering  adds 
cost  and  development  time  to  the  project  without  adding  value.  The  project  manager  is  responsible  for  product 
development  within  a  schedule  that  is  market  driven.  Schedule  is  the  driving  factor  and  an  unreliable  product 
will  be  "pushed  out  the  door"  to  meet  the  schedule.  Problems  will  be  mitigated  through  good  servicing  or,  in 
severe  cases,  will  be  solved  through  redesign. 

The  personnel  of  the  quality  assurance  organization  are  often  asked  to  participate  in  project  development 
reviews,  but  their  involvement  is  principally  from  a  Quality  Assurance  perspective  rather  than  a  reliability 
perspective.  In  many  cases  these  reviews  provide  the  only  opportunity  for  representatives  of  the  Reliability 
Departrment  to  present  and  discuss  reliability  issues  relating  to  the  project  development. 

Reliability  testing,  when  performed,  is  performed  at  the  system  level  and  is  restricted  to  growth  testing. 
Vendor-supplied  components  and  products  are  selected  based  on  performance  parameters  but  are  not  subject  to 
acceptance  testing.  Testing  of  vendor  products  is  an  area  that  a  few  of  the  reliability  engineers  are  strongly 
encouraging  be  increased  or  imposed.  Environmental  Stress  Screening  (ESS)  is  used  if  problems  are  uncovered 
in  field  service  (i.e.,  in  a  reactive  mode)  but  is  not  a  standard  part  of  the  production  process. 

The  "Robust  Design"  theme  is  promoted  throughout  the  company.  The  reliability  engineers  expressed  the 
concern  that  the  project  managers  believe  that  if  they  produce  a  robust  design,  reliability  need  not  be  addressed. 
This  attitude  is  one  reason  R&M  tasks  are  not  implemented.  We  concluded  that  the  relationship  between 
R&M  and  robust  design  is  not  understood. 

The  company  is  attempting  to  limit  the  number  of  its  suppliers,  and  preferred  vendors  lists  are  kept  for 
the  primary  components.  It  was  imclear  what  criteria  are  used  for  developing  these  lists. 

Summary  of  Interview  of  Company  C  RepresentativefsI.  Company  C's  products  are  sold  to  private 
individuals  and  industrial  users.  Sales  are  made  through  independent  dealers  and  distributorships  and,  in  a  few 
cases,  directly  to  the  end  user.  Products  are  normally  standard  or  a  custom  integration  of  standard  components, 
although  a  few  genuinely  custom  products  are  developed  from  time  to  time. 

Design  and  reliability  (D&R)  specifications  for  catalog  products  and  components  are  developed  internally 
using  warranty  data,  market  and  customer  surveys,  feedback  from  the  sales  and  service  organization,  and  some 
failed-part  an^yses.  Customers  usually  specify  system-level  design  performance  and  features  for  build-to-order 
products.  They  do  this  by  including  "sophisticated"  clauses  in  the  contract.  These  clauses  impose  monetary 
penalties  for  failing  to  meet  downtime  and  specify  service  call  rates,  uptime  ratio,  and  availability.  Internal 
specifications  include  Mean  Time  Between  Failure  (MTBF),  unit  failure  rates,  service  call  rates,  design  life, 
shipping  and  transportation  requirements,  and  operating  envelope.  Also  addressed  are  regulatory  requirements 
imposed  by  governments.  Selected  customers  are  asked  to  help  vahdate  D&R  specifications. 

Concurrent  engineering  is  implemented  using  product  development  teams.  These  teams  are  under  the 
direction  of  a  general  manager  who  is  responsible  for  all  aspects  of  the  product.  This  responsibility  includes 
the  requirement  to  budget  for  warranty  costs.  The  motivation  for  designing  for  reliability  is  to  make 
production  start-up  easy,  reduce  failure  rates  (and  warranty  costs),  and  reduce  the  number  of  "dead-on-arrivals." 
Reliability  engineers  are  part  of  these  teams.  A  centralized  reliability  organization  supports  these  team 
engineers  by  providing  training,  tools,  and  general  requirements  and  managing  product  test  laboratories.  Many 
of  the  same  tasks  from  MIL-STD-785  are  used  during  the  design  but  there  are  significant  differences  in  their 
application.  For  example,  little  documentation  or  review  accompanies  the  implementation  of  tasks.  The 
procedures  for  the  tasks  are  tailored  for  the  company's  use  (predictions  are  made  using  their  own  data  base. 


D-2-4 


testing  methods  are  company-developed,  pass-fail  testing  is  seldom  used,  and  FMEAs  are  used  to  prioritize 
areas  of  risk).  Mechanical  component  reliability  predictions  are  made  using  probabilistic  structural  methods. 
Reliability  engineers  also  use  finite  element  analysis,  boundary  element  analysis,  rapid  prototyping,  and 
knowledge-based  engineering  tools. 

For  purchased  items,  a  two-level  supplier  program  is  used.  For  major  or  critical  components,  the 
con^any  maintains  a  "partnership”  with  two  certified  suppliers.  For  other  conqwnents,  only  one  supplier  is 
maintained  and  not  all  of  these  suppliers  will  be  certified.  Audits  are  used  to  certify  suppliers  and  certified 
suppliers  must  "sign  off”  on  applications  of  their  parts  or  conqxinents  (i.e.,  concur  that  their  part  or  conqwnent 
will  not  be  overstressed  in  the  planned  operating  environment).  Incoming  inspections  are  never  made  of 
products  from  certified  suppliers, 

Summaiy  of  Interview  of  Company  D  Representative(s).  Company  D's  products  are  sold  directly  to 
original  equipment  manufactureres  (OEMs).  Standard  and  custom  products  are  developed;  custom  products  are 
developed  for  major  updates  to  OEM  systems  and  for  entirely  new  system  models.  Customers  stipulate  the  cost 
and  schedule  requirements  for  a  product.  Few,  if  any,  design  specifications  are  provided  by  customers. 
Performance  requirements  are  usually  functional  in  nature  and  stated  in  the  context  of  existing  or  prior  products 
(i.e.,  we  need  a  product  similar  to  the  one  used  in  our  1992  system  but  with  5%  better  performance).  The 
design  specification  and  design  are  defined  and  evolve  together. 

The  company  receives  an  abundance  of  performance  data  from  end  users  through  a  central  OEM 
computer.  However,  the  data  are  not  precise  and  contain  "noise"  due  to  inaccurate  data  recording.  Failed 
products  are  returned  from  the  OEMs’  service  department  that  repaired  the  system  and  are  analyzed  to  determine 
the  cause  of  failure.  Some  electronic  units  have  a  50%  retest  OK  rate.  In  general,  tear-down  analysis  of  failed 
components  results  in  10-90%  "no  trouble  found."  A  failure  in  a  product  or  process  is  defined  as  not  meeting  a 
requirement 

Products  are  developed  through  a  hierarchy  of  Business,  Product,  and  Project  Teams  that  guide  the 
product  through  a  four-ph^  development  process:  Concept  Phase  (0),  Design  Validation  Phase  (1),  Process 
Validation  (2),  and  Production  (3).  The  Business  Team  has  high-level  representatives  from  sales,  marketing, 
engineering,  and  manufacturing.  The  Business  Team  has  cost  and  schedule  objectives,  but  it  was  unclear  from 
the  interview  how  funding  and  budgeting  are  handled.  Each  Business  Team  has  several  Product  Teams.  Each 
Product  Team  can  have  several  Project  Teams,  each  focusing  on  a  specific  cortqx)nent  or  design  element.  The 
membership  of  each  Project  Team  consists  of  designers,  design  assurance  engineers,  and  manufacturing 
representatives.  Each  team  works  at  a  different  level  of  product  assembly. 

The  design  assurance  engineer  is  responsible  for  reliability  and  related  assurance  disciplines.  Usually, 
there  is  no  specification,  and  "system"  requirements  are  frequently  given  verbally  by  the  customer. 
Requirements  are  not  allocated  to  lower  levels  of  assembly.  No  formal  process  has  been  established  for  the 
translation  of  requirements  to  specifications.  Specifications  evolve  from  the  various  analyses,  especially 
sensitivity  and  "what  if  analyses,  and  testing  performed  during  the  design  and  development  process.  No 
specific  reliability  requirements  are  institutionalized,  and  no  central  reliability  organization  exists.  The  vision 
is  for  a  central  design  assurance  office  to  provide  matrix  management  support  to  individual  product  teams.  The 
incentive  to  "do  R&M"  is  that  they  believe  providing  products  with  high  R&M  provides  a  competitive 
advantage,  and  they  want  to  avoid  the  costs  of  failures.  A  failure  is  defined  as  not  meeting  the  requirement. 

Product  Teams  use  a  variety  of  reliability  tools,  including  Fault  Tree  Analysis,  FMECA,  and 
predictions.  Predictions  are  useful  in  developing  the  specification.  Service  analysis,  criticality  analysis,  and 
warranty  prediction  (i.e.,  predicting  warranty  costs  and  frequency)  are  also  used.  Past  warranty  ^ta  are  used  to 
assess  the  evolving  design.  No  requirements  or  tools  are  mandated  by  corporate  policy;  guidelines  are  available 
to  all  product  and  project  teams. 

Until  recently,  little  if  any  documentation  was  kept  on  reliability  tasks.  More  is  being  kept,  primarily 
because  of  the  change  to  a  competitive  environment.  This  documentation  is  delivered  with  the  product.  The 
company  has  general  procedures  for  various  activities. 

Product  development  is  constrained  by  some  Federal  and  state  goverment  regulations.  In  some  cases, 
state  regulations  are  more  severe  than  those  of  the  Federal  Government. 
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Supplier  assurance  activities  are  centered  on  applying  internal  procedures  externally.  A  mix  of 
certification  and  incoming  inspection  is  used,  with  products  from  certified  suppliers  receiving  the  least  amount 
of  inspection.  The  objective  of  the  company  is  to  reduce  the  number  of  suppliers  and  to  establish  long-term 
relationships  (i.e.,  long-term  contracts)  with  the  remaining  suppliers.  It  is  also  the  company’s  objective  to 
eliminate  dl  incoming  inspection  by  woridng  with  suppliers  to  certify  processes  and  by  establishing  a  level  of 
confidence  and  trust.  Suppliers  are  certified  through  an  auditing  procedure.  The  compare  has  started 
vahdating  its  own  parts  and  is  beginning  to  validate  supplier  parts.  Performance  of  parts  is  tracked  over  time 
and  must  be  satisfactory  for  a  supplier  to  remain  certified. 

Other  information  and  opinions  offerred  were: 

•  Movement  toward  design  teams  and  concurrent  engineering  is  essential. 

•  Design  teams  must  be  given  a  reliability  perspective  (as  well  as  a  cost,  schedule,  and  performance 
perspective)  eaiiy  in  a  development  program. 

•  Mditaiy  procurement  is  marred  by  excessive  paperwork  and  by  duphcative  effort. 

•  Failure  modes  and  effects  analyses  should  be  used  in  developing  test  plans  and  evaluating  safety. 

•  Failure  mechanisms  must  be  analyzed  to  develop  accelerated  test  factors. 

•  Use  of  Bayesian  statistics  in  assessing  rehabihty  needs  to  be  explored  and  developed. 

•  Emphasis  in  testing  should  be  on  validation  testing  using  accelerated  test  factors. 

Summary  of  Interview  with  Company  E  RepresentativefsV  This  company’s  products  are  used  by  a  wide 
variety  of  government,  private,  and  pubUc  customers.  The  con:q)any  has  a  larger  share  of  the  world  market  than 
any  competitor.  A  finite  number  of  models  are  produced,  but  a  wide  range  of  options  is  available  to  customers. 

Commercial  customers  do  not  specify  rehability  or  specific  rehability  tasks.  They  select  a  product  based 
on  their  needs  and  come  to  the  conipany  based  on  its  reputation  and  the  historical  performance  of  its  products. 
As  long  as  the  reliability  is  within  an  acceptable  range,  cost  or  performance  is  the  main  customer  concern. 
Some  government  regulations  must  be  met  before  the  product  may  be  sold;  however,  these  do  not  impose  the 
equivalent  of  MIL-STD-785  or  MIL-STD-78 1 .  The  company  characterizes  the  performance  of  its  own  products 
and  those  of  its  competition  to  determine  the  level  of  reliability  needed  to  meet  its  strategy  of  building  the 
world's  most  reliable  product.  In  determining  the  reliability  required,  the  company  distinguishes  between 
mission  and  logistics  reliability. 

Commercial  products  are  warranted  for  a  maximum  calendar  time  or  operating  hours  (the  latter  is 
prorated  after  a  threshold  number  of  hours).  Concurrent  warranties  are  provided  by  major  suppliers,  but 
customers  need  only  go  to  the  company  for  all  warranty  questions  and  problems.  Extended  warranties,  up  to  a 
fiill  life  warranty,  are  available.  Design  problems  are  covered  beyond  the  basic  warranty  period,  essentially  for 
the  life  of  the  product.  Operators  sometimes  do  some  of  their  own  maintenance  During  the  warranty  period, 
the  company  reimburses  customers  for  repairs  made  by  or  paid  for  by  them.  For  serious  problems,  customers 
call  the  company’s  service  center  which  will  take  appropriate  action,  including  sending  a  technician.  The 
warranty  period  is  used  to  collect  field  performance  data  Ibat  are  fed  back  to  the  design  engineers.  Items  that 
fail  or  wear  out  during  the  warranty  period  are  analyzed  to  determine  the  cause  of  failure  or  wear-out.  Results  of 
failure  analysis  are  used  to  update  and  revise  design  standards.  In  addition,  a  team  meets  weekly  to  address 
customer  service  problems.  The  team  is  composed  of  representatives  from  Product  Support,  project 
management,  engineering,  quality,  R&M,  manufacturing,  and  warranty  administration. 

Multi-disciplined  design-build  teams  (DBTs)  are  used  for  new  designs.  The  DBT  approach  is  the 
company’s  way  of  implementing  the  concept  of  concurrent  engineering.  A  product  development  committee 
assigns  a  team  leader,  who  may  be  from  engineering,  manufacturing,  etc.  The  DBTs  have  representation  from 
engineering,  manufacturing,  R&M,  quahty,  and  Product  Support.  A  complete  CAD/CAM  system  supports  the 
DBTs,  which  address  both  the  product  design  and  manufacturing  processes.  According  to  the  individual 
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interviewed,  the  company’s  number  one  strategy  for  competing  in  the  world  mariietplace  is  to  make  its 
product  the  world’s  most  reliable.  In  this  context,  mechanical  fatigue  is  an  important  design  concern.  The 
basic  philosophy  for  designing  a  reliable  product  is  to  make  progressive,  evolutionary  changes  to  improve 
successive  models.  Seldom  does  the  next  model  represent  a  leap  in  technology,  performance,  or  design.  Field 
experience  is  heavily  relied  on  to  identify  both  design  deficiencies  and  preferred  designs.  New  materials  or 
technology  are  incorporated  only  after  extensive  testing.  In  designing  for  reliability,  the  DBTs  use  techniques 
and  approaches  not  dissimilar  in  title  from  those  often  imposed  on  military  contractors: 

•  failure  modes  and  effects  analysis  (required  prior  to  release  of  any  drawings) 

•  allocation  and  prediction  (including  use  of  the  David  Taylor  mechanical  reliability  prediction 
models) 

•  failure  reporting  and  corrective  action  system 

•  dedicated  reliability  testing  (data  from  all  testing  are  used,  however) 

In  many  ways,  however,  these  techniques  and  approaches  are  implemented  for  commercial  customers  in  a  way 
that  differs  from  the  implementation  for  government  customers.  Much  of  the  required  format,  approval,  and 
documentation  associated  with  the  imposition  of  a  "government  standards  is  absent  in  the  commercid  arena. 

The  effect  on  reliability  of  proposed  design  changes  to  parts  is  evaluated  by  the  DBTs  using  a  metric 
called  the  Reliability  Characterization  Number  (RCN).  This  company-developed  metric  is  a  cumulative 
measure  of  the  effects  of  a  design  change  on  reliability,  safety,  and  cost.  By  using  the  RCN,  proposed  design 
changes  are  subjected  to  a  mini-cost^enefit  analysis  before  adoption.  Specifics  on  the  metric,  how  it  is 
measured  or  predicted,  etc.,  were  not  made  available. 

Another  key  to  designing  a  reliable  product  is  to  analyze  failures,  especially  field  failures,  to  determine 
the  root  causes  of  failure  and  then  to  use  the  resulting  information  to  make  engineering  and  manufacturing 
changes  and  to  update  and  revise  corporate  practices.  Standardized  practices  (e.g.,  design  manuals  which 
specify  design  "rules,"  including  R&M  rules)  are  used  throughout  the  company,  but  the  word  standardized  may 
not  mean  exactly  the  same  thing  as  for  a  company  making  only  products  for  the  government.  In  other  words, 
much  more  tailoring  and  project-unique  application  of  a  procedure  exist  than  are  normally  provided  for  in 
government  contracts.  Equally  noteworthy,  however,  is  that  the  company’s  design  manuals  often  reference 
MIL-STDs.  The  MIL-STDs  are  considered  good  guidance  but  are  not  invoked  as  stringently  as  is  done  on 
military  contracts. 

Based  on  the  company’s  assessment  of  the  level  of  reliability  required,  reliability  goals  are  set  at  the 
product  level  and  allocated  down  to  lower-levels.  Predictions  are  used  by  the  DBTs  to  evaluate  progress  toward 
meeting  these  goals.  The  "best"  source  data  is  used  in  making  these  predictions,  preferably  historical  data  or 
knowledge  of  similar  products 

Another  indication  of  its  emphasis  on  reliability  is  the  company’s  Reliability  Board.  Constituted  of 
vice  president/director-level  representatives  from  all  programs  and  ch^ed  by  the  Vice  President  of  New  Product 
Development,  this  board  meets  every  two  weeks  to  ensure  that  the  reliability  levels  expected  by  customers  are 
being  achieved;  to  find  solutions  to  customer,  design,  or  manufacturing  problems,  and  to  identify  processes 
needing  improvement.  Issues  include  needed  training,  parts  selection  and  application,  design  rules,  and 
manufacturing  processes. 

The  company  tries  to  hire  either  mature,  experienced  R&M  engineers  or  young  engineers  who  practical 
design  experience.  Reliability  training  done  in-house  is  limited  and  given  primarily  for  design  engineers. 

The  company  strives  to  establish  long-term  relationships  with  suppliers.  Five-year  production  options 
are  often  used  to  establish  these  relationships  and  to  foster  a  team  concept.  A  supplier  qualification  program  is 
used  that  includes  site  surveys,  process  qualification,  process  specifications,  and  product  qualification.  The 
stringency  of  the  qualification  process  varies  with  the  criticality  of  the  parts  or  supplies  being  purchased. 
Second-sourcing  is  used  whenever  possible.  MTBF  requirements  are  included  for  some  purchased  products. 
An  MTBF  guarantee  is  used  to  enforce  MTBF  requirements  and  product  data  are  used  to  measure  compliance 
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with  the  guarantee.  Emerging  or  identified  reliability  problems  due  to  supplier  parts  are  investigated  and 
solved  by  a  DBT  working  with  the  supplier  in  a  team  approach.  Penalties  for  failing  to  meet  a  guaranteed 
MTBF  include  providing  extra  spares,  repairing  failed  units  at  no  cost,  etc. 

In  closing,  the  individual  added  some  personal  observations,  including  the  following  two:  commercial 
reliabihty  plans  and  programs  are  more  cost-effective  than  the  government  equivalents  and  yield  results  that  are 
at  least  as  good;  and,  specifying  MIL-STD  parts  does  not  guarantee  good  parts. 

Summary  of  Interview  with  Company  F  RepresentativersV  The  products  of  this  company  account  for  a 
majority  of  the  world  market.  Customers  are  involved  in  the  development  of  new  products;  they  can  observe 
production  and  perform  a  rigorous  inspection  of  every  design  and  operating  detail.  Customers  usually  have  few 
reliability  requirements,  and  these  are  stated  in  operational  terms.  The  specification  value  of  each  reliability 
requirement  is  developed  through  technical  discussions  held  between  the  company  and  customers.  The  product 
has  a  that  includes  a  reliability  guarantee.  Failure  to  meet  the  guarantee  results  in  a  penalty  for  the  company. 
The  guaranteed  rehability  begins  at  a  baseline  value  and  gradually  increases  each  year  to  some  "mature”  value, 
thereby  anticipating  reliability  growth 

Four  major  process  areas  are  designated  as  key  to  the  success  of  the  company  and  constitute  the 
acquisition  cycle  of  the  company's  products.  These  areas  are: 

•  Business  Acquisition  (sales,  marketing,  preliminary  design) 

•  Definition  (engineering  design  and  drawings) 

•  Production 

•  Support  (after-sales  service  provided  by  the  Customer  Service  Division) 

The  company’s  goal  in  re-engineering  its  processes  is  to  reduce  build  time,  reduce  engineering  and 
manufacturing  costs,  and  to  maintain  a  technological/performance  edge  over  its  competition.  Integrated 
Design/Build  Teams  (IDBTs)  are  the  company’s  approach  to  implementing  a  concurrent  engineering 
philosophy.  A  CAD  system  facilitates  the  design  team  approach.  Each  IDBT  is  headed  up  by  a  design 
engineer  and  has  the  following  representation: 

•  Customer  Service  •  Operations  (manufacturing) 

•  Toohng  •  Material 

•  Reliability  Engineering  •  Safety 

•  Customer 

The  primary  role  of  the  reliability  engineers  on  the  IDBTs  is  to  collect  "facts  and  data"  from  operational 
experience,  analyze  the  facts  and  data,  and  provide  lessons-leamed  to  the  team.  This  experiential  information  is 
used  in  trade  studies  to  avoid  past  problems  through  design  changes.  The  reliabihty  engineers  also  do  the  more 
traditional  tasks,  such  as  performing  a  FMECA.  No  "proof  documentation  is  supplied  to  customers.  The 
only  documentation  of  reliability  tasks  maintained  is  that  required  by  government  regulation  and  what  is 
actually  needed  to  get  the  job  done.  The  design  tools  and  activities  used  are: 

•  Accelerated  testing  (temperature  and  vibration). 

•  Markov  analysis  (transition  from  one  state  to  another). 

•  Robust  architecture  and  design. 

•  Fault  Tree  Analysis  (performed  on  "very  complex"  systems). 

•  FMECA,  usually  on  redundant  systems.  FMECA  is  on  the  critical  design  path.  The  use  of 
FMECAs  is  not  yet  widespread  but  is  gaining  acceptance. 

•  FRACAS  (in-service  and  manufacturing). 

•  No  reliability  or  maintainability  predictions  (but  see  later  paragraph  on  suppliers). 
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DOE  (used  to  identify  key  characteristics). 


The  objectives  of  reliability  engineering  are: 

•  Ensure  safety 

•  Ensure  that  the  product  meets  government  requirements 

•  Reduce  availability  costs:  determine  where  costs  are  and  then  design  to  reduce  or 
eliminate. 

•  Use  field  data  to  determine  (predict)  failure  frequency  and  to  identify  failure  causes. 

The  company  has  recently  embraced  a  corporate  approach  to  quality  based  on  the  use  of  SPC  on  the 
manufacturing  floor;  and  the  identification  of  key  product  characteristics  (DOE  is  identified  as  a  key  technique 
for  this  purpose).  As  a  result  of  this  quality  approach,  hardware  variability  (design  and  manufacturing)  control 
is  a  major  concern.  Continuous  Quality  Improvement  (CQI)  is  an  active  goal.  They  realized  that  their  initial 
CQI  efforts  were  focused  on  inproving  rework  processes.  The  emphasis  is  now  changing  to  that  of  eliminating 
rework.  In  an  attempt  to  remain  "world-class”,  the  company  benchmarks  its  competitors 

The  company  emphasizes  the  importance  of  long-term,  constructive  relationships  with  suppliers.  In 
general,  the  company  attempts  to  remove  constraints  from  suppliers  and  to  help  "keep  them  in  business"  as 
long  as  they  keep  the  company  happy  (i.e.,  provide  high  quality  supplies  that  allow  the  company  to  meet 
customer  needs,  and  remain  competitive  ,  from  a  cost,  quality,  and  technological  perspective).  The  emphasis  is 
on  qualifying  processes  rather  than  on  qualifying  parts  or  product. 

Few  mandates  are  given  to  suppliers.  Accelerated  testing  guidelines  are  given  to  suppliers,  and  they  are 
required  to  provide  a  report  to  the  respective  IDBT(s)  that  identifies  the  approach  used  for  accelerated  testing 
and  the  results  of  that  testing.  However,  they  are  not  directed  how  to  perform  the  accelerated  testing. 
Suppliers  are  usually  required  to  provide  some  prediction  of  reliability.  The  company’s  preference  is  that  they 
use  actual  data  to  do  this,  but  MIL-HDBK-217  predictions  are  acceptable. 

The  individual  provided  other  opinions  and  anecdotal  information  during  the  discussion.  Those 
opinions  and  information  are  provided  here,  without  comment,  as  accurately  as  possible. 

•  Reliability  engineering  is  transitioning  from  its  historical  role  to  one  of  facilitator  and 

educator/tiainer. 

•  The  acquisition  process  in  the  commercial  world  is  less  structured  than  in  the  government.  This 
allows  the  commercial  world's  process  to  be  more  timely,  less  expensive,  and  more  effective. 
Commercial  designers  have  almost  total  autonomy  with  little  (if  any)  customer  directioa 

•  The  long-term,  partner-oriented  relationships  being  established  in  the  commercial  world  between 
the  customer  and  the  contractor,  and  between  the  contractor  and  its  subcontractors,  is  difficult  to 
imagine  in  the  govenunent  procurement  world. 

•  Military  specifications,  standards,  and  handbooks  are  very  useful  resources  but  they  must  not  be 
mandated.  Mandating  their  use  and  lack  of  tailoring  drives  up  costs,  relieves  the  contractor  of 
some  responsibility  for  the  outcome,  and  produces  rsults  with  limited  utility. 

•  "Black"  programs  for  the  military  more  closely  approximate  the  commercial  world.  Security 
restrictions  force  streamlined  procedures  and  team  approaches.  "Black"  programs  are  thus  more 
cost-effective  and  timely  than  open  programs. 

•  MIL-HDBK-217  and  its  underlying  assumptions,  such  as  a  constant  failure  rate,  are  not  the 
problem.  The  problems  lie  in  the  fact  that  over  the  years  they  have  promoted  fallacious  ideas. 
These  fallacious  ideas  are  that  failures  are  inevitable,  that  equipment  fails  at  random  times,  and 
that  reliability  can  only  be  improved  by  miming  electronics  cooler  or  by  using  better  parts. 
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•  Failures  are  preventable  during  the  service  life  of  an  item. 

Summary  of  Interview  with  Company  G  Representativefsl  Company  G  has  three  principal  product 
divisions.  The  first  two  divisions  sell  to  both  government  and  commercial  customers  while  the  third  sells  only 
to  commercial  customers.  The  company’s  products  (presently  numbering  some  several  hundred  active  products) 
are  primarily  custom-designed  products  rather  than  standard  catalogue  items. 

A  failure  is  defined  after  delivery  as  "something  that  the  customer  identifies  as  not  woiidng  and  is 
subsequently  confirmed  by  test."  Before  shipment,  a  failure  is  defined  as  any  condition  that  prevents  the 
product  from  meeting  acceptance  criteria. 

Due  to  the  sophisticated  nature  of  the  test  equipment,  repair  is  "depot-level."  It  is  typically  performed 
under  a  nominal  warranty  (calendar  years);  the  trend,  however,  is  toward  longer  warranty  periods.  Most 
customers  specify  a  warranty  period  and  an  MTBF.  When  a  failure  occurs  during  the  warranty  period,  the  user 
does  the  initial  diagnosis  and  returns  the  item  in  question  to  the  company  for  repair.  Most  customers  request  a 
subsequent  failure  analysis  report  on  the  failure.  Thus,  an  effective  failure  data  base  has  been  established  by  the 
company.  Warranty  costs  are  tracked  and  incorporated  into  the  pricing  structure.  The  company  does  not  offer 
service  contracts;  after-warranty  repair  costs  are  charged  to  the  customer  on  a  case-by-case  basis. 

Individual  product  design  teams  are  used  except  where  small,  multiple-product  teams  are  more 
applicable.  Each  of  the  three  product  divisions  has  a  different  reliability  structure.  An  inqx)itant  fact  is  that 
most  customers  are  knowledgeable  regarding  rehability,  especially  with  respect  to  risk.  Some  design  teams  are 
assigned  dedicated  reliability  engineers,  while  other  design  teams  have  no  specific  reliability  engineering 
support.  In  these  cases,  the  design  engineer  (who  has  also  been  trained  in  reliability)  is  responsible  for  the 
product  rehability. 

Each  division  manager  is  responsible  for  both  the  quality  and  reliability  of  the  division's  products 
during  design,  production,  and  warranty.  Reliability  is  important  to  the  division  managers  because  it  affects 
both  sales  (more  reliable  products  sell  better)  and  warranty  costs  (more  rehable  products  reduce  the  cost  of  a 
warranty).  Both  cost  (price,  warranty  costs,  etc.)  and  risk  (loss  of  customers,  habUity,  warranty  costs  in  excess 
of  projected,  etc.)  are  the  key  concerns.  Customers  are  involved  in  the  product  specification  and  design,  and 
they  actively  audit  the  company's  manufacturing  process. 

The  keys  to  reliability  are: 

•  Using  proven  technology 

•  Using  processes  proven  off-line 

•  Performing  life  testing  of  their  products 

•  Performing  step-stress  testing  to  determine  margins 

More  attention  is  typically  paid  to  commercial  design  than  to  a  comparable  government  design  because: 

•  The  required  tum-around  time  for  commercial  products  is  shorter. 

•  Commercial  customers  are  more  demanding  when  it  comes  to  field  performance  and  the  degree  to 
which  they  hold  the  company  responsible  for  field  performance. 

•  In  the  case  of  government  customers,  you  only  have  to  "build  to  spec."  Commercial  customers 
do  not  tell  you  how  to  do  the  job  (i.e.,  few  if  any  specs)  and,  therefore,  assume  no  responsibility 
for  the  performance  of  the  product. 
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As  stated  earlier,  most  customers  specify  a  warranty  period  and  an  MTBF.  Reliability  predictions,  based 
upon  either  field  data  (preferred)  or  MIL-HDBK-217^  are  used  to  estimate  MTBF,  but  they  are  not  used  to 
niake  engineering  judgments. 

The  product  design  teams  determine  which  specific  reUability  tasks  will  be  performed.  Only  those  tasks 
that  are  determined  to  be  "value-added”  will  be  performed.  A  value-added  task  is  one  that  prevents  a  product 
from  being  returned  (for  rework  before  delivery,  or  by  the  customer  after  delivery)  or  one  required  by  the 
customer.  FMECAs  and  fault  tree  analysis  are  usually  performed  only  when  specifically  requested  by  the 
customer.  FMECAs,  when  they  are  done,  are  done  "off-line”  rather  than  using  the  company’s  CAD/CAM 
system.  Automated  MIL-HDBK-217  reliability  predictions  are  used.  DOE  is  an  important  task  used  to 
structure  testing. 

Environmental  testing,  safety,  and  other  issues  related  to  reliability  are  the  responsibility  of 
organizations  other  than  the  reliability  group.  The  degree  to  which  coordination  and  integration  of  efforts 
among  reliability,  environmental  testing,  safety,  quality^,  etc.,  varies  from  division  to  division  and  from 
design  team  to  design  team.  The  issue  of  coordinating  and  integrating  related  disciplines  is  one  that  those 
interviewed  identified  as  needing  improvement. 

The  company  has  foimd  that  their  products  are  sensitive  to  the  effectiveness  of  the  manufacturing 
process.  Including  manufacturing  in  the  design  process  has  been  an  important  part  of  the  company’s  design 
team  approach.  Design  verification  testing  also  is  intended  to  bring  manufacturing  into  the  design  process. 
Incremental  process  improvement  in  the  product/process  development  is  emphasized.  This  process 
improvement  frequently  involves  step  stress  testing.  Testing  is  based  upon  understanding  the  operating 
environment  and  good  engineering  judgment.  As  already  stated,  DOE  is  used  to  help  structure  testing. 

With  the  significant  reductions  in  the  number  of  quality  and  reliability  personnel  which  the  company  has 
recently  experienced,  internal  reUability  education  has  become  a  major  objective  of  the  reliability  organization. 
Formal  reliability  training,  including  design  of  experiments,  has  been  established  for  reliability  engineers, 
design  engineers,  and  test  engineers  and  has  been  presented  to  between  over  a  hundred  employees. 

Customers  are  now  very  concerned  with  part  selection  and  control.  The  focus  is  on  qualifying  famiUes 
of  parts  for  specific  appUcations  while  auditing  the  distributors.  The  intent  is  to  encourage  suppliers  to  "buy 
into  a  design"  rather  than  mandating  requirements.  The  company  has  observed  better  response  from 
subcontractors  by  specifying  "when,  what,  and  how  much"  than  by  specifying  "how  to." 

Don't  mandate  specifications  and  standards.  Military  specifications  and  standards  have  significant  utility 
as  resources.  The  problem  is  how  they  are  implemented;  their  value  is  lost  when  they  are  mandated.  This 
problem  is  especially  true  as  the  requirements  are  flowed  down  by  the  contractor  to  the  subcontractors.  While 
there  may  be  some  flexibility  between  the  government  and  the  prime  contractor,  this  flexibiUty  is  quickly  lost 
as  the  detailed  requirements  are  levied  on  successive  layers  of  subcontractors.  A  meaningful  flow-down  of  the 
customer's  needs  is  needed,  not  detailed  "how  to  do  it"  direction. 

By  specifying  a  lot  of  detail  and  mandating  specifications,  standards,  and  the  use  of  specific  handbooks, 
the  government  takes  much  of  the  responsibility  off  the  contractor,  who  "merely  builds  to  spec  as  directed." 
By  specifying  only  results  and  leaving  the  "how  to"  to  the  contractor,  the  customer  places  the  burden  of 
responsibihty  for  the  design  and  the  field  results  on  the  contractor. 

Summar>Lof  Interview  with  Company  H  Representative's).  Company  H  produces  industrial  components 
for  OEMs.  The  company  is  organized  into  four  major  businesses.  The  responses  to  our  interview  questions 
reflect  operations  within  the  largest  of  those  businesses  and  would  not  necessarily  reflect  the  operations  of  the 
others. 


^The  interviewees  stated  that  it  is  their  experience  that  failure  rates  based  upon  field  data  are  typically  lower 
than  MIL-HDBK-217  based  failure  rates. 

^The  (^uahty  Office  is  responsible  for  the  quality  of  both  design  and  manufacturing  processes  but  performs  only 
an  audit  functioa 
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The  product  is  best  characterized  as  semi-custom,  because  even  though  each  EOM  has  his  own  unique 
requirements,  there  is  also  a  large  degree  of  commonalty  among  the  different  products.  OEMs  add  additional 
equipment  to  the  product  to  produce  a  complete  “system”. 

The  major  incentive  for  the  company  to  consider  reliabihty  is  that  they  are  essentially  marketing  a  ”throw- 
away"  component.  Rework  or  repair  is  extremely  expensive.  Products  are  typically  sold  with  a  multiple-year 
warranty.  However,  because  the  products  exhibit  virtually  no  susceptibility  to  wear-out  mechanisms,  longer 
warranty  periods  would  be  of  Uttle  value  to  the  customer. 

Much  of  the  company’s  product  development  process  has  only  been  recentiy  formalized.  A  lack  of  formal 
procedures  and  consistency  led  the  compare  to  institute  changes  to  formalize  the  process  and  to  mandate  certain 
procedures.  Concurrent  engineering  is  not  yet  fully  implemented  but  is  a  goal. 

A  project  manager,  an  engineer  from  the  Engineering  Department,  is  assigned  at  the  beginning  of  initial 
development  of  a  product.  This  engineer  is  responsible  for  all  aspects  of  the  design,  including  reliability.  At 
the  point  at  which  the  design  goes  into  production,  responsibility  shifts  to  manufacturing,  although  an 
engineering  haison  continues  to  provide  support.  Engineering  oversees  testing,  but  Quality  Assurance  (QA)  is 
responsible  for  conducting  the  tests.  QA  ^so  analyzes  field  failures  and  provides  the  results  to  Engineering. 
Maiketing  is  responsible  for  warranty  costs  and  repairs. 

The  company’s  approach  to  achieving  rehability  emphasizes  component  design.  This  emphasis  takes  the 
form  of  the  use  of  design  rules,  adequate  design  margins,  and  proper  derating  in  designing  components  and  in 
qualification  testing  of  all  key  components.  The  product  test  program  is  based  upon  a  variation  matrix  to 
determine  what  parameters  are  to  be  selected  for  control  (i.e.,  key  design  characteristics).  These  key  design 
characteristics  are  identified  by  each  individual  component  engineer.  As  is  frequently  the  case  in  specialized 
component  manufacturing,  the  characterization  and  the  control  of  the  critical  parameters  of  some  materials  used 
in  the  product  is  the  most  important  contributing  factor  to  its  ultimate  performance  and  reliability.  SuppUer 
control  is,  consequently,  an  important  part  of  achieving  high  reliability.  The  specific  testing  approach  for  a 
given  component  is  developed  by  the  component  engineer  who  uses  experience  to  balance  the  cost  of  testing 
against  the  cost  of  failure.  Cost  of  failure  is  measured  not  only  in  the  economic  impact  but  in  diminished 
reputation  and  loss  of  customer  confidence. 

A  product  typically  has  a  ten  to  twenty  year  design  life,  and  the  market  life  of  a  given  design  is  three  to 
five  years.  Some  specific  safety-related  government  requirements  are  levied  on  the  product.  Informal  FMEAs 
are  now  done  to  the  extent  needed  to  verify  that  there  are  no  single  point  failures.  A  Fault  Tree  Analysis  is  also 
performed  to  satisfy  the  UL  requirement  for  a  safety  report.  UL,  Canadian  Standards  Association  (CSA),  and 
ISO  sj^ifications  and  standards  are  used  rather  than  MIL-SPECs  or  MIL-STDs.  In  the  case  of  user  safety,  the 
interviewee  stated  that  the  UL  requirements  are  frequently  more  stringent  than  the  comparable  MIL- 
SPECs/STDs. 

The  product’s  reliability  requirement  is  specified  in  terms  of  an  MTBF;  there  are  also  maintainability 
requirements  dealing  with  accessibility,  fault  isolation,  and  the  tools  and  personnel  required  to  perform  a 
specific  repair.  Demonstrated  compliance  with  these  requirements  is  based  on  the  analysis  of  field  data,  and 
this  analysis  is  delivered  to  the  customer.  The  OEM  is  responsible  for  repairs,  with  support  from  the 
company’s  field  service  which  is  provided  at  no  cost  during  the  warranty  period  (for  warranted  items,  of  course) 
and  on  a  case-by-case  basis  thereafter.  The  company  does  not  deal  in  maintenance  contracts.  Field  failures 
occur  primarily  in  the  equipment  added  by  the  OEM,  not  in  the  company’s  product. 

Purchased  critical  components  are  qualification  tested,  and  critical  suppliers  must  be  qualified  based  on 
their  processes  and  past  performance.  Suppliers  are  controlled  primarily  through  performance  specifications. 
Material  suppliers  provide  documentation  of  material  properties  with  the  product.  In  cases  where  specialized 
testing  by  a  supplier  would  be  prohibitively  expensive,  the  company  accepts  materials  that  have  passed  the 
supplier's  standard  tests.  The  company  then  either  does  more  extensive  sample  testing  or  ensures  that  the 
design  margins  for  the  component  in  which  the  material  is  used  are  sufficient  to  allow  for  variations  in  the 
material.  Although  the  company  does  a  limited  amount  of  sample  testing  for  all  suppliers,  the  major  burden  of 
proof  usually  is  placed  directly  upon  the  supplier.  Alternative  sources  are  maintained  to  provide  a  competitive 
environment,  but  long-term  supplier  relationships  are  also  encouraged. 
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Other  information: 


•  the  interviewee(s)  would  not  advocate  doing  away  with  DOD  standards  and  handbooks. 

•  the  company  has  difficulty  obtaining  field  performance  data  after  the  warranty  period.  They  are 
working  with  the  OEMs  to  improve  this  situation. 

Summary  of  Interview  with  Company  I  Representativefsy  In  its  world-wide  facilities,  this  company 
manufactures  millions  of  its  product  each  year  and  sells  to  OEMs,  retailers,  and  end  users). 

The  product  comes  with  a  warranty  (calendar  year),  the  length  of  which  depends  on  the  market  and 
customer.  A  company  service  organization  provides  toll-free  telephone  support  to  end  customers.  Defective 
products  can  be  returned  during  the  warranty  period  to  service  points.  Defective  products  are  repaired  at  the 
service  point  if  the  problem  is  simple  to  correct.  Complicated  repairs  are  done  at  one  of  the  company’s 
manufacturing  facihties. 

Product  requirements  come  from  three  major  sources:  market  research,  competitive  equipment  analysis, 
and  focus  groups  of  end  users.  Some  custom  requirements  may  also  be  stipulated  by  OEMs.  Those 
interviewed  made  it  clear  that  the  company’s  focus  is  on  total  customer  satisfaction,  so  product  performance 
requirements  (what  is  needed)  are  externally  driven.  As  will  be  discussed  in  the  next  section,  the  means  of 
meeting  these  requirements  (how  to  do  it)  are  internally  driven,  and  customers  do  not  play  a  role  in  defining 
these  means.  Some  govenunent  and  other  external  forces,  ISO  9000,  for  example,  do  dictate  some 
requirements.  For  the  most  part,  however,  the  company  focuses  on  deriving  product  and  process  specifications 
and  requirements  from  their  understanding  of  the  customers’  performance  requirements  and  their  evaluation  of 
the  competition. 

For  large  developments,  a  “focused  team”  is  created  with  members  from  design,  manufacturing,  quality, 
marketing,  and  service.  This  focused  team  represents  the  company’s  implementation  of  the  concept  of 
concurrent  engineering.  Reliability  engineers  are  assigned  to  the  focused  teams.  However,  it  was  emphasized 
that  the  reliability  engineers  provide  technical  assistance  and  guidance;  the  design  engineers  do  the  worst  case 
analysis,  prediction,  and  other  analyses.  When  asked  if  the  design  team  leader  or  another  individual  was 
explicitly  held  accountable  for  long-term  success  (e.g.,  has  responsibility  for  budgeting  for  the  warranty),  it  was 
stated  that  the  company’s  culture  had  progressed  to  a  point  where  “threats”  were  not  needed  to  ensure  quality 
up-front  Instead,  the  motivation  for  doing  a  quality  job  and  producing  a  reliable  product  comes  from  knowing 
the  performance  of  the  competitions’  products  and  the  now-instinctive  desire  to  satisfy  the  customer.  However, 
all  analyses  are  subject  to  peer  review  by  a  group  not  associated  with  the  product  in  question.  Long-term  goals, 
technic^y  and  financially,  are  derived  from  the  company’s  assessment  of  customers  and  competition. 

Based  on  the  assessment  of  the  competition  and  the  needs  of  the  customers,  product  requirements, 
including  rehability,  are  set.  All  requirements  are  documented  in  a  Product  Certification  Book  that  is  approved 
by  the  Quality  organization.  Also  included  in  the  Product  Certification  Book  are  specific  reliability  tasks,  such 
as  Worst  Case  Analysis,  MIL-HDBK-217  parts  count  prediction,  and  Accelerated  Life  Testing  (ALT). 
Reliability  goals  are  usually  stated  as  a  fraction  of  a  percent  of  failures  per  month  for  the  product  population 
(i.e.,  a  fafture  rate).  Closely  associated  with  reliability,  quahty  is  measured  in  defects  per  unit.  To  achieve  the 
reliability  goal,  the  team  uses  MIL-HDBK-217  parts  coimt  predictions,  worst  case  analysis,  and  extensive 
development  testing.  Root  cause  analysis  is  important  to  the  company  and  the  development  testing  is  intended 
to  reveal  any  weaknesses  under  all  conditions  that  could  reasonably  be  expected  during  the  life  of  the  product. 
Testing  is  ALT  and  is  required  for  all  products.  Testing  is  performed  by  the  company’s  ALT  Reliability 
Engineering  Group  to: 

•  subject  products  to  extreme  environments  similar  to  what  the  customers  may  encounter 

•  accelerate  the  life  of  the  product  by  compressing  approximately  five  years  of  use  into  five  weeks  of 
testing 

•  uncover  design  flaws  so  development  engineers  can  improve  design  margins 

•  precipitate  parts  failures  and  provide  information  on  potential  weaknesses  to  suppliers 
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•  identify  process  problems  so  manufacturing  engineers  can  develop  efficient  and  controlled 
processes  before  production  begins 

ALT  includes  subjecting  products  to  dropping,  temperature  extremes  (hot  and  cold),  temperature  cycling 
(gradual  change  from  hot  to  cold  and  cold  to  hot),  temperature  shock  (sudden  change  from  hot  to  cold  and  cold 
to  hot),  electrostatic  discharge,  exposure  to  dust,  vibration,  and  humidity.  Besides  being  used  during  the 
development  of  a  product,  ALT  is  required  to  be  performed  quarterly  on  a  sample  of  in-production  products. 
This  requirement  ensures  that  any  clmges  in  the  process  or  other  anomalies  are  detected  early  and  corrected. 
ALT  is  also  used  to  compare  the  company’s  product  with  those  of  the  competition.  For  these  comparisons,  a 
company  product  and  a  competitor’s  product  are  placed  in  an  ALT  cycle  simultaneously  under  identical  test 
conditions.  Finally,  the  design  of  the  ALT  is  one  input  to  Certification  Testing  (discussed  in  more  detail  in  a 
succeeding  paragraph).  Major  new  products  are  also  subjected  to  Beta  site  testing,  it  was  stated  that,  based  on 
the  ALT,  that  the  designers  have  a  high  confidence  that  the  reliability  goal  has  been  met;  this  point  was  not 
elaborated  on  by  those  interviewed. 

When  questioned  how  the  ALT  test  parameters  are  determined  to  ensure  all  possible  failure  modes  are 
stimulated,  the  interviewee(s)  cited  experience  as  the  key  basis  for  the  parameters.  Asked  if  an  FMEA  or 
FMECA  was  conducted,  the  reply  was  an  emphatic  no  (except  when  required  by  the  customer.  It  is  believed 
that  an  FMEA  adds  paper  but  no  value  to  the  engineering  design  process.  Further  probing,  however,  suggested 
that  company  engineers  must  do  something  analogous  to  an  FMEA  to  ensure  that  they  have  accounted  for  all 
failure  modes.  For  example,  in  further  discussion,  the  interviewee(s)  criticized  the  “formal  structure  and 
documentation”  associated  with  an  FMEA.  It  is  the  opinion  of  the  interviewers  that  the  interviewee(s)  was 
equating  FMEA  with  MIL-STD-I629.  Prior  interviews  of  individuals  from  other  con^anies  revealed  that  they 
too  were  against  the  “overhead”  associated  with  the  formal  FMEA  as  described  in  the  standard.  However,  these 
individuals  stated  that  their  analyses  incorporate  or  include  some  of  the  essential  elements  of  an  FMEA  as  part 
of  their  efforts  to  know  which  failure  modes  are  applicable  to  their  design.  Given  the  company’s  emphasis  on 
identifying  failure  modes  and  root  causes,  it  is  likely  that  the  essential  elements  of  an  FMEA  are  incorporated 
in  their  reliability  engineering  efforts. 

Before  being  released  for  shipment,  a  product  must  be  approved  by  Quality  (“no  approval,  no  ship”). 
In  the  case  of  a  new  model,  part  of  the  approval  entails  a  product  successfully  passing  a  series  of  tests  called 
Certification  Testing  (CT);  CT  is  distinct  from  ALT,  but  many  of  the  tests  are  similar.  CT  is  specifically 
performed  on  samples  of  a  new  product  to  confirm  that  in  all  respects,  the  product  represents  Best-in-Class 
quality.  The  tests  are  prescribed  in  a  Prototype  Product  Certification  Guidelines  document  created  by  the  ALT 
Reliability  Engineering  Department.  The  tests  consist  of: 

•  sudden  impact  testing 

•  endurance  testing 

•  package  drop  testing  (ensures  packaging  prevents  damage  during  shipping) 

•  transport  simulation 

•  electrical  certification 

The  company  has  a  form  of  a  failure  reporting  and  corrective  action  system,  at  least  during  the  warranty 
period.  All  service  actions  are  recorded,  and  the  data  used  to  track  field  reliabihty.  As  stated  earlier,  other  than 
making  simple  repairs,  customers  return  all  failed  items  to  the  service  organization.  Accordingly,  the  company 
engineers  are  able  to  analyze  failures  in  detail.  Regular  technical  meetings  are  held  to  evduate  the  field 
reUability,  review  problems,  and  identify  actions  to  improve  the  product. 

Software  quality  is  receiving  more  attention  because  the  software  is  becoming  the  main  “component”  of  a 
product.  Software  qu^ity  is  measured  in  terms  such  as  defects  per  1000  lines  of  code.  The  company’s  goal  is 
to  be  certified  at  the  Software  Engineering  Institute’s  level  3  by  1995. 
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The  company’s  approach  to  suppliers  can  be  summarized  in  eight  points: 

•  suppliers  considered  Partners  for  Progress 

•  all  purchased  items  are  dual-sourced 

•  suppliers  are  expected  to  meet  same  requirements  as  the  company  (with  the  exception  of  ISO  9000 
-  suppliers  are  not  required  to  conform  to  ISO  9000) 

•  processes  must  be  certified 

•  limited  number  of  suppliers 

•  suppliers  must  meet  the  Quality  System  Review  Guidelines  to  be  on  the  con^any’s  Qualified 
Suppliers  List  (QSL) 

•  training  of  suppliers  is  essential 

•  failure  to  meet  standards  results  in  a  supplier  being  eliminated  from  the  QSL 

•  suppliers  are  asked  to  give  a  time  frame  in  which  they  will  apply  for  the  Malcolm  Baldrige 
Award 

Two  points  bear  discussion.  First,  by  dual  sourcing  all  purchased  items,  the  company  has  allowed  for 
periods  of  “surge”  production,  will  not  be  forced  to  halt  production  if  a  process  problem  appears  in  a  supplier 
process,  and  has  the  latitude  to  immediately  “fire”  a  supplier  who  fails  to  measure  up  to  the  company’s 
standards.  Second,  suppliers  are  expected  to  meet  all  the  same  standards  as  is  the  company  itself. 

In  addition  to  the  material  already  presented,  the  following  comments  and  opinions  were  offered: 

•  Government  requirements  (imposed  on  its  contractors)  are  prescriptive;  customer  requirements 
should  be  descriptive 

•  the  problem  with  standards  of  any  kind,  regardless  of  the  source,  is  that  they  foster  bureaucracy 
and  stifle  innovation 

•  dual-use  R&M  standards  as  advocated  by  the  G- 11  Committee  of  SAE,  would  be  of  doubtful 
value  to  the  company  but  may  be  useful  to  companies  with  a  less  mature  quality  culture 

•  the  standards  of  value  and  concern  to  the  company  involve  interoperability  within  an  industry 

•  other  than  MIL-HDBK-217,  no  military  or  government  standards  and  handbooks  related  to  R&M 
aroused 

Summaiv  of  Interview  with  Company  J  Representative tsV  This  company’s  products  are,  for  the  most 
part,  standard  catalog  items,  but  some  customizing  of  products  is  done.  Products  are  sold  directly  to  end  users 
throughout  the  US,  Europe,  and  South  America.  Customers  do  not  specify  product  performance  (including 
reliability).  The  company  "markets”  the  high  reliability  of  its  products.  Products  are  shipped  with  operating 
and  service  manuals. 

A  failure  is  as  any  time  the  product  does  not  perform  its  function.  Products  having  a  microprocessors 
perform  self-test  functions  during  power-up  and  operation.  The  products  are  maintained  by  the  user  who 
removes  and  replaces  failed  subassemblies  (equivalent  to  Air  Force  Line  Replaceable  Units)  and  ships  the  failed 
subassemblies  back  to  the  company’s  service  department.  The  service  department  provides  support  to  the  users 
and  repairs  the  failed  subassemblies.  In  the  case  of  recurring  or  critical  failures,  failed  components  (i.e..  Shop 
Replaceable  Units)  and  Field  Service  Reports  are  subjected  to  failure  analysis.  Most  failures  occur  in 
electronics,  specifically  in  the  interconnects.  Shipping  and  handling  is  the  severest  environment  to  which  the 
products  are  subjected. 
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The  company  is  extremely  conscious  of  the  need  for  a  safe  product  -  this  consciousness  is  the  driving 
factor  behind  most  of  what  they  do.  Products  are  developed  using  a  six-phase  approach  called  the  Product 
Development  Process  (PDP).  The  six  phases  are:  Idea  Generation,  Product  Planning,  Design,  Validation, 
Production  and  Launch,  and  Customer  Delivery. 

Some  of  the  key  reliability  tasks  performed  are  predictions,  using  MIL-HDBK-217  with  a  mix  of  field 
data;  a  parts  count  prediction  to  identify  problem  areas;  hazard  analysis;  life  testing;  and  demonstration  testing. 
The  use  of  a  hazard  analysis  has  recently  been  introduced  and  is  replacing  the  use  of  an  FMEA.  A  hazard 
analysis  is  performed  from  the  top  down  and  from  the  bottom  up  and  at  the  board  level  and  at  the  component 
level.  The  purpose  of  this  and  related  analyses  is  to  identify  critical  components.  It  is  similar  to  an  FMEA 
and  includes  use/misuse  analysis.  The  design  team  (DT)  concept,  i.e.,  concurrent  engineering,  is  used  to 
develop  all  products.  Many  of  the  tasks  are  based  on  DoD  guidelines  (e.g.,  MIL-STD-785,  MIL-STD-781,  and 
MIL-STD-1629)  but  are  tailored  to  meet  the  company’s  needs. 

Customers  do  not  specify  a  required  reliability.  An  MTBF  goal  was  developed  by  evaluating  the  state  of 
technology  and  benchmarking  the  competitor's  products  (in  which  competitors'  products  are  tom  down  and 
studied).  Marketing  then  set  the  MTBF  goal  as  necessary  to  lead  the  competition.  Field  data  and  a  MIL- 
HDBK-217  prediction  are  used  to  verify  that  the  goal  is  being  achieved  on  current  products.  Historical  field 
data  and  parts  count  analysis  of  old  equipment  are  used  to  determine  historical  reliabilify.  Failure  analyses  of 
failed  products  are  used  to  develop  design  requirements.  Currently,  most  failures  occur  in  intercormects  on  the 
electronics  board.  Based  on  the  failure  analyses  performed,  the  boards  are  redesigned  to  eliminate  these  failures. 
New  designs  are  overstressed  to  failure  to  determine  how  and  where  failures  occur  and  to  determine  the 
robustness  of  the  design.  Boards  are  subjected  to  thermal  cycling  and  are  thermally  mapped  (using  thermistors) 
to  qualify  them  prior  to  design  release.  (Outside  test  agencies  are  used  to  perform  thermal  testing.)  New 
designs  are  "pre-qualified"  using  a  parts  count  prediction.  For  critical  components,  all  tests  are  zero-failure.  A 
reliability  demonstration  is  not  really  needed  due  to  the  extensive  component-level  testing,  but  a  system-level 
reliability  demonstration  is  perform^  using  a  company-developed  test  design.  It  is  an  accelerated  test  and  is 
based  on  the  Chi  Square  method  to  minimize  the  number  of  test  articles.  Life  testing  is  also  performed. 

Reliability  costs  are  not  budgeted.  It  was  unclear  from  our  discussions  how  a  DT  manager  accounts  for 
the  costs  of  reliability  tasks.  The  value  of  reliability  tasks  does  not  seem  to  be  questioned,  probably  because  of 
the  potential  life-threatening  consequences  of  failure.  Organizationally,  the  reliability  and  product  assurance 
offices  have  joint  responsibilities  for  reliability.  These  responsibilities  are  allocated  as  follows: 

RELIABILITY  PRODUCT  ASSURANCE 

Design  Good  Manufacturing  Practices  (GMPs) 

Specification  Field  failures 

Analysis 

Test 

The  company’s  products  are  subject  to  government  regulations.  In  addition  to  govermnent  regulations, 
the  company's  products  must  comply  with  UL  requirements.  International  Electrotechnical  Commission  (lEC) 
(Europe)  requirements,  and  ISO  9000  standards. 

Suppliers  are  chosen  based  on  performance  and  reputation.  The  company's  Quality  Assurance  (QA) 
organization  checks  suppliers  to  ensure  that  their  QA  processes  are  acceptable.  Suppliers  do  all  end  testing;  the 
company  performs  no  incoming  inspections  other  than  visual.  Suppliers  must  be  qualified.  The  MTBF 
requirement  is  not  allocated  to  suppliers. 

Other  information  included: 

•  Commercial  industiy  needs  practical,  down-to-earth,  common  sense  guides,  not  theory 

•  MIL  standards  and  specifications  form  the  basis  for  much  of  what  commercial  industiy  does 

•  MIL  standards  and  specifications  must  be  tailored,  not  become  ends  in  themselves,  and 
developing  organization  must  have  latitude  in  their  application 
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Summary  of  Interview  with  Company  K  Representativefsl  Company  K  produces  a  wide  range  of  related 
products.  Products  are  standard,  off-the-shelf  (catalog)  items  with  options  available  to  meet  specific  customer 
needs.  No  customized  products,  in  the  true  sense  of  the  term,  are  made. 

Design  specifications  are  developed  based  on  market  research  and  surveys,  knowledge  of  competitors' 
equipment  performance,  and  technology.  Benchmarking  is  not  a  key  activity  because  benchmaiking  data  are 
often  too  late  (i.e.,  competition  has  already  gained  a  competitive  edge).  However,  some  performance 
benchmaiking  is  used  to  determine  the  performance  of  the  con^any’s  products  relative  to  competitors'  products. 
Competitive  benchmarking,  in  which  competitors'  products  are  tom  down,  is  not  performed  because  few,  if 
any,  differences  exist  in  most  areas. 

Customers  are  offered  a  selection  of  one  of  several  levels  of  after-sales  support  (after-sales  support  is  not 
the  responsibility  of  the  designers  or  product  development  teams  but  a  service  division).  Basically,  ftiese  levels 
differ  by  the  guaranteed  response  time  in  the  event  of  a  problem.  The  cost  of  the  various  levels  is  the  same 
regardless  of  the  cost  and  type  of  system  purchased.  Customers  call  a  toll-free  number  that  is  answered  by 
company  personnel.  Based  on  their  assessment  of  the  problem,  they  dispatch  a  service  representative  to  fix  the 
equipment  (usually  a  remove  and  replace  action).  Service  representatives  are  called  "change  engineers,"  and 
service  is  often  subcontracted  out  within  the  United  States.  In  Europe,  service  is  done  primarily  through 
messengers.  European  customers  who  have  a  problem  with  a  product  call  a  toll-free  number,  the  company 
representative  assesses  the  problem  based  on  information  provid^  by  the  customer  and  dispatches  a  messenger 
with  a  replacement  part. 

From  its  records  on  maintenance  of  its  products  while  under  warranty,  the  conq)any  tracks  response 
times  and  parts  usage  fairly  accurately  for  products  sold  to  end  users.  However,  reliability  and  failure  data  are 
difficult  to  obtain.  This  difficulty  results  from  the  lack  of  complete  configuration  control  after  the  product  is 
sold  and  an  inability  to  accurately  track  operating  hours.  It  is  also  difficult  to  obtain  aity  data  on  products  sold 
through  resellers,  although  quality  assurance  forms  are  used  to  collect  feedback  from  OEMs.  These  difficulties 
are  not  significant  to  the  company  because  its  goal  is  not  to  determine  root  causes  of  failures  but  to  do  trending 
and  identify  problem  areas. 

Customers  are  primarily  interested  in  performance  and  product  availability.  The  rather  recent  emphasis 
on  availability  is  creating  a  new  concern  by  company  designers  in  the  maintainability  of  their  products. 
Selected  customers  are  asked  to  participate  in  evaluating  designs  of  current  products  and  projected  specifications 
for  new  products.  Customers,  DoD,  and  major  OEMs,  regularly  audit  the  company's  processes  and  qualify  its 
products.  Considerable  amounts  of  design  and  test  data  are  made  available  to  these  customers  in  conjunction 
with  these  audits  and  qualification  activities. 

The  company  has  a  New  Product  Introduction  process  consisting  of  seven  phases:  Exploration;  Initiation 
of  Approval;  Specification,  Planning,  and  Design;  First  Prototype;  Pre-Pilot  (Beta  testing);  Pre-Launch 
(qualification  of  manufacturing  processes);  Sustaining  Production;  and  End  of  Life.  The  cycle  time  from 
Exploration  to  Pre-Launch  for  a  completely  new  product  ranges  from  8  months  to  2  years.  Design  teams  are 
designated  to  develop  products  under  the  leadership  of  project  managers  (i.e.,  this  process  is  the  company’s 
implementation  of  a  concurrent  engineering  approach). 

The  interviewee(s)  defined  failure  as  noncompliance  with  the  specification  definition  of  operational 
performance.  The  focus  of  reliability  engineering  within  the  company  is  on  qualification  of  the  design. 
Product  reliability  is  the  responsibihty  of  the  project  managers  and  design  teams.  The  reliability  organization 
provides  support  and  guidance  to  the  design  teams.  Project  managers  are  evaluated  and  awarded  bonuses  on  the 
basis  of  the  number  of  units  sold:  time-to-market  and  time-to-volume  are  the  drivers.  At  first  glance,  with  this 
emphasis  on  schedule,  one  might  conclude  that  reliability  is  not  a  key  concern  of  the  project  managers. 
However,  manufacturing,  which  has  responsibility  for  after-sales  rehability,  is  a  member  of  the  design  team  and 
must  sign  off  on  the  product  release.  The  design  teams  seek  consensus  on  the  design  approach,  requirements, 
and  activities/tools  used  in  the  design. 

Marketing  research  is  used  to  develop  the  product  specifications.  Requests  for  quote  (RFQs)  received 
from  potential  customers  usually  include  a  reliability  requirement  (either  an  MTBF,  an  availability  or  an 
allowable  downtime  requirement).  In  its  response  to  an  RFQ,  the  company  includes  a  MIL-HDBK-217 
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prediction  to  support  its  claim  regarding  the  product  reliability.  If  the  customer  challenges  the  prediction,  the 
company’s  approach  is  to  educate  the  customer  regarding  the  meaning,  underlying  assumptions,  and  limitations 
of  the  prediction.  Because  the  company's  products  are  catalog  items,  the  reliability  cannot  be  "changed"  to 
quiet  a  customer's  concern.  The  greatest  source  of  this  concern  are  the  claims  by  the  company’s  competitors  of 
having  equivalent  products  with  higher  reliability,  claims  that  are  based  on  other,  sometimes  questionable 
means  of  predicting  or  measuring  reliabihty.  The  company  uses  MIL-HDBK-217  rather  than  other  methods 
because  the  MIL-HDBK-217  methodology  is  universally  known  or  accepted  and  addresses  key  design  factors 
(for  example,  the  Bellcore  methodology  is  not  well  accepted  overseas  and  does  not  adequately  address  thermal 
considerations).  These  MIL-HDBK-217  failure  rate  predictions  are  considered  conservative,  usually  1.2  to  3.6 
lower  than  what  will  be  experienced  in  service.  In  fact,  the  Japanese  will  multiply  the  MIL-HDBK-217  MTBF 
prediction  by  3  and  specify  the  result  as  the  rehability  value  to  be  demonstrated  (the  Japanese  usually  require  a 
complete  rehability  demonstration).  Based  on  predictions,  the  reliability  staff  recommends  design  changes  to 
design  teams,  develops  test  plans,  and  so  forth. 

Along  with  reliability  prediction,  thermal  analysis  and  evaluation  (using  a  thermal  scanner  on 
prototypes)  are  key  reliability  tasks.  Neither  a  FMECA  nor  a  fault  tree  analysis  (FT A)  is  performed:  they 
became  "victims"  of  downsizing  (i.e.,  not  feasible  given  the  present  staff).  Reliability  growth  is  just  now 
being  introduced  into  the  company.  It  is  used  if  a  product  does  not  meet  the  reliability  requirement. 
Environmental  evaluation  (humi^ty,  temperature,  vibration,  shock,  etc.)  is  done  by  an  organization  other  than 
the  reliabihty  department  to  ensure  that  a  product  meets  minimum  military  and  lEC  specifications.  Accelerated 
testing  (a  modified  Highly  Accelerated  Life  Test  [HALT]  method)  is  used.  Early  involvement  in  the  design 
concept  and  development  phases  is  considered  essential  to  the  success  of  a  product  in  the  marketplace.  Some 
use  is  made  of  DOE  (Design  of  Experiments)  to  determine  key  design  characteristics  and  to  develop  test  plans. 
Step  stress  testing  is  first  done  using  operational  levels  of  stresses  to  prove  out  the  design;  it  is  then  performed 
at  higher  levels,  until  the  product  fails,  to  determine  design  margins.  No  maintainability  analysis  has  been 
done  in  the  past;  it  is  now  being  added  in  light  of  customer  emphasis  on  availability.  The  current 
maintainability  requirement  was  subjectively  and  somewhat  arbitrarily  determined.  The  value-^ded  and  cost  of 
reliability  activities  (i.e.,  the  return  on  investment)  is  difficult  to  quantify.  Activities  are  selected  by  design 
teams  based  on  engineering  judgment  and  experience.  Activities  and  testing  are  documented  to  satisfy  internal 
requirements  and  business  partrrers  or  to  allay  the  concerns  of  certifying  bodies,  such  as  UL. 

Software  rehability  is  a  major  concern  and  is  considered  the  major  issue  for  the  future.  Its  importance 
lies  in  the  fact  that  most  hardware  becoming  extremely  reliable.  Therefore,  hardware  reliability  will  become  a 
moot  point  within  5  years;  thus  reliability  will  become  totally  software  driven.  Little  distinction,  in  terms  of 
definition  or  approach,  is  made  between  hardware  and  software  rehability.  The  key  is  to  emphasize  the  design 
process  and  on  reducing  the  frequency  with  which  the  software  fails  to  perform  as  required  (or  planned). 

The  reliability  staff  provides  guidelines  to  the  supplier  engineers  (who  are  part  of  the  design  teams) 
regarding  suppher  specifications  for  reliabihty.  These  supplier  engineers  track  supplier  comphance.  Two  levels 
of  suppliers  are  used:  Preferred  Suppliers  and  Partner  Suppliers.  The  production  processes  of  Preferred 
Suppliers  are  qualified  by  the  company.  In  the  case  of  Partner  Suppliers,  the  company  becomes  very  involved 
with  the  details  of  the  supphers’  designs.  Extensive  documentation  regarding  the  performance  and  testing  of 
supplier  products  is  required.  The  company  ensures  that  its  suppliers'  equipment  is  electrostatic  discharge 
(ESD)  qualified. 

Other  information  offered  included: 

•  The  creativity  of  design  engineers  is  hmited  by  standard  processes  and  mandated  procedures. 

•  The  company's  management  priorities  are,  in  order:  price,  performance,  then  rehabihty. 

•  Because  the  cost  of  ownership  is  difficult  to  measure,  price  still  dominates. 
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TERMS  AND  DEFINITIONS 


AAJVQ  Association  for  the  Advancement  of  Medical  Instrumentation 

Acceptance  Testing  Any  testing  required  by  customers  prior  to  their  acceptance  of  a  product.  Can  be  done 
before  committing  to  production  (see  RQT)  or  during  production  to  “ensure”  that  the 
designed  reliability  is  not  degraded  by  the  manufacturing  processes  (see  PRAT). 

ASME  American  Society  of  Mechanical  Engineers 


Benchmarking 


CAD/CAM 

CAFTA 

Competitive 

Benchmarking 


CSA 


Benchmarking  is  the  continuous  process  of  measuring  products,  services,  and  practices 
against  the  toughest  competitors  or  those  companies  recognized  as  industry  leaders. 
(David  T,  Kearns,  Former  Chief  Executive  Officer,  Xerox  Corporation) 

Computer-aided  design/computer-aided  manufacturing 
Computer-aided  Fault  Tree  Analysis 

Benchmarking  done  against  direct  competitors  {Benchmarking  by  Robert  C.  Camp, 
ASQC  Quality  Press,  Milwaukee,  WI,  1989.) 

Sometimes  referred  to  as  strategic  or  corporate  benchmarking. 

Canadian  Standards  Association 


Demonstration 

Testing 

DOE 


DOT 

DTIC 


Testing  done  to  demonstrate  that  a  specified  reliability  has  been  made  (i.e.,  determine 
compliance).  Acceptance  testing  (see),  PRAT  (see),  and  RQT  (see)  are  demonstration 
tests. 

Design  of  Experiments.  A  branch  of  applied  statistics  dealing  with  planning, 
conducting,  analyzing,  and  interpreting  controlled  tests  to  evaluate  the  factors  that 
control  the  value  of  a  parameter  or  group  of  parameters.  {Quality  Progress,  February 
1992,  page  22) 

Department  of  Transportation 
Defense  Technical  Information  Center 


EPA 

ESS 


FAA 

FCC 

FMEA 


FMECA 

FRACAS 

FTA 


Functional 

Benchmarking 


Environmental  Protection  Agency 

Environmental  Stress  Screening.  A  series  of  tests  conducted  under  environmental 
stresses  to  disclose  weak  parts  and  workmanship  defects  for  correction.  (MIL-STD- 
721C) 

Federal  Aviation  Administration 
Federal  Communications  Commission 

Failure  Modes  and  Effects  Analysis.  A  procedure  by  which  each  potential  failure  mode 
in  a  system  is  analyzed  to  determine  the  results  or  effects  thereof  on  the  system  and  to 
classify  each  potential  failure  mode  according  to  its  severity. 

Failure  Modes,  Effects,  and  Criticality  Analysis.  See  FMEA. 

Failure  Reporting  and  Corrective  Action  System.  A  closed-loop  system  for  reporting 
failures,  analyzing  the  failures  to  determine  cause,  and  recording  the  corrective  action 
taken.  (MIL-STD-785B) 

Fault  Tree  Analysis.  An  analytical  method  to  identify  the  effects  of  faults  on  a  system. 
It  is  a  top-down  approach  in  contract  to  the  FMECA  which  is  a  bottom-up  approach. 
{Reliability  Toolkit,  RL,  April  1993) 

Benchmarking  of  a  function  using  for  comparison  a  company  in  a  dissimilar  industry 
(i.e.,  not  a  direct  competitor).  {Benchmarking  hy  Camp) 

Also  defined  as  benchmarking  of  functions,  rather  than  strategic  measures  of  company 
performance,  using  either  direct  competitors  or  companies  in  dissimilar  industries. 
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GAO 

Growth  Testing 

IBC 

lEC 

IITRI 

Internal 

Benchmarking 

ISO 

Maintainability 


MIL-HDBK 

MIL-SPEC 

MIL-STD 

Normative 

Benchmarking 

NUREG 

OASD  (ES) 
OSHA 

Physics  of  Failure 

POC 

PRAT 

Quality 


QualiHcation  Test 

RAAF 

R&M 

RAC 

Reliability 


General  Accounting  Office 
See  RGT. 

International  Benchmarking  Clearinghouse 
International  Electrotechnical  Commission 
Ilhnois  Institute  of  Technology  Research  Institute 

Benchmarking  of  a  function  by  a  company  organization  using  another  operating  unit 
within  the  company  for  comparison. 

International  Standards  Organization 

The  measure  of  the  ability  of  an  item  to  be  retained  in  or  restored  to  a  specified 
condition  when  maintenance  is  performed  by  personnel  having  specified  skill  levels, 
using  prescribed  procedures  and  resources,  at  each  prescribed  level  of  maintenance  and 
repair.  (MIL-STD-721C) 

Military  Handbook 

Military  Specification 
Military  Standard 

Benchmarking  done  for  a  cooperating  group  of  companies  by  a  third-party  consultant. 
Results  and  data  are  aggregated  (they  cannot  be  traced  or  attributed  to  a  specific 
company)  and  then  distributed  to  all  companies 

Nuclear  Regulatory  Commission 

Office  of  the  Assistant  Secretary  of  Defense,  Economic  Security 
Occupational  Safety  and  Health  Administration 

An  approach  to  developing  robust  designs  and  manufacturing  processes  based  on 
understanding  the  root-cause  failure  mechanisms. 

Point  of  Contact 

Production  Reliability  Acceptance  Test.  See  Acceptance  Testing. 

The  degree  of  excellence  possessed  by  a  product,  service  ,  or  other  output  of  a  business 
activity  or  business  process.  (Anon) 

Fitness  for  use.  (J.  M.  Juran) 

Meeting  the  customer’s  requirements  the  first  time  and  every  time.  (Federal  Quality 
Institute) 

Cost  to  society  and  robustness.  (Genichi  Taguchi) 

See  RQT. 

Royal  Australian  Air  Force 
Rehability  and  Maintainability 

Reliability  Analysis  Center.  A  DoD-sponsored  Information  Analysis  Center 
administratively  managed  and  funded  by  the  Defense  Technical  Information  Center.  The 
RAC’s  technical  areas  of  responsibihty  are  rehability,  maintainability,  and  quality. 

(1)  The  duration  or  probabihty  of  failure-free  performance  under  stated  conditions. 

(2)  The  probability  that  an  item  can  perform  its  intended  function  for  a  specified 
interval  under  stated  conditions.  (For  non-redundant  item,  this  is  equivalent  to 
definition  (1).  For  redundant  items,  this  is  equivalent  to  the  definition  of  mission 
reliabihty.)  (MIL-STD-721C) 
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RGT 


Reliability  Growth  Testing.  Testing  performed  to  improve  reliability  by  uncovering 
and  correcting  deficiencies  in  design  or  manufacture. 

RQT  Reliability  Qualification  Testing.  Testing  performed  for  the  purpose  of  demonstrating, 

or  measuring,  the  reliability  of  an  item.  The  testing  is  used  to  determine  contractual 
comphance  with  pre-established  acceptance-reject  criteria.  (MIL-HDBK-781) 

SIC  Standard  Industrial  Code 

Strategic  Benchmarking  of  direct  competitors  using  corporate-level  indicators.  (Kersi  F.  Munshi, 

Benchmarking  Consultant) 

Also  called  corporate  or  competitive  benchmarking. 

TAAF  Test,  Analyze,  and  Fix.  See  RGT. 

Tailoring  The  inclusion,  deletion,  and  modification  of  specific  requirements  of  a  military  standard 

or  specification  to  match  program  and  product  requirements  and  constraints. 

TQM  Total  Quality  Management 

UL  Underwriter’ s  Laboratory 

WCCA  Worst-case  Circuit  Analysis.  An  analysis  performed  to  evaluate  the  simultaneous 

existence  and  effect  of  all  unfavorable  parameter  tolerances  on  a  circuit.  {Reliability 
Toolkit,  RL,  1993) 

WSIG  Weapon  Support  Improvement  Group 
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RAC  Product  Order  Form  — . . 

Ordering  Title  Page#  U.S.  Non-US  Qty.  Item 

Code  Price  Price  Total 


Data  Publications 


NPRD 

Nonelectronic  Parts  Reliabilitv  Data 

3 

$195 

$215 

ECDS 

Environmental  Characterization  Device  Sourcebook 

3 

■lEsa 

$120 

RDSC-2 

The  Reliability  Sourcebook  -  "How  and  Where  to  Obtain  R&M  Data  and 
Information" 

3 

$50 

$60 

VZAP 

Electrostatic  Discharae  Susceptibilitv  Data 

3 

$195 

$215 

NONOP-1 

Nonoperatina  Reliability  Databook 

3 

$50 

$60 

FMD 

Failure  Mode/Mechanism  Distributions 

3 

$100 

BIkM 

Application  Guides 


CPE 

Reliabilitv  Toolkit:  Commercial  Practices  Edition 

4 

$35 

SLEA 

Service  Life  Extension  Assessment 

4 

$50 

$60 

SOAR-2 

Practical  Statistical  Anaivsis  for  the  Reliabilitv  Enaineer 

4 

$60 

BENCH 

Benchmarkina  Commercial  Reliabilitv  Practices 

4 

RMST 

Reliabilitv  &  Maintainabilitv  Software  Tools 

4 

$50 

TEST 

Testabilitv  Design  and  Assessment  Tools 

4 

RTMG 

RAC  Thermal  Manaaement  Guidebook 

5 

$75 

$85 

NPS 

Mechanical  Applications  in  Reliabilitv  Enaineerina  *Price  Reduced* 

5 

■1^9 

QREF 

RAC  Quick  Reference  Guides 

5 

SOAR-6 

ESD  Control  In  the  Manufacturina  Environment 

5 

$50 

SOAR-4 

Confidence  Bounds  for  System  Reliability 

5 

$50 

WHDK 

New  Weibull  Handbook 

6 

$79 

$94 

Reliable  Application  of  Components 


PSAC 

Parts  Selection,  Application  and  Control 

6 

$75 

EOES 

EOS/ESD  Guidelines 

6 

CAP 

Reliable  Application  of  Capacitors 

6 

PEM2 

Reliable  AoDlication  of  Plastic  Encapsulated  Microcircuits 

6 

MCM 

Reliable  Application  of  Multichip  Modules 

6 

$50 

$60 

MFAT-1 

Microelectronics  Failure  Anaivsis  Techniques:  A  Procedural  Guide 

7 

$70 

Km 

MFAT-2 

GaAs  Microcircuit  Characterization  and  Failure  Analysis  Techniques: 

A  Procedural  Guide 

7 

$50 

$60 

MFAT1  &2 

Combined  set  of  MFAT-1  and  MFAT-2 

7 

o 

o 

$120 

QML 

Qualified  Manufacturer's  List:  New  Device  Manufacturing  and 
Procurement  Techniaue 

7 

$50 

$60 

GAAS 

An  Assessment  of  Gallium  Arsenide  Device  Quality  and  Reliability 

7 

$60 

ATH 

Analoa  Testina  Handbook 

7 

$100 

$120 

PRIM 

A  Primer  for  DoD  Reliability,  Maintainability,  Safety  and  Logistics 
Standards  ‘Price  Reduced* 

mm 

$50 

$60 

Quaiitv  Improvement 


SPAT 

Software  Enaineerina  Process  Group  Handbook 

8 

Bm 

$40 

BPRQ 

Business  Process  Reenaineerina  for  Quality  Improvement 

8 

Bisa 

$85 

TQM 

TQM  Toolkit 

8 

KiinB 

SOAR-7  1 

A  Guide  for  Implementina  Total  Quaiitv  Manaaement 

8 

$75 

SOAR-8  i 

Process  Action  Team  Handbook  ‘Price  Reduced* 

8 

$30 

Computer  Products 


CART 

RAC  Computer-Aided  Reliability  Trainina  Course 

9 

$300 

$340 

217N2 

MIL-HDBK-217F,  Notice  2  (Macintosh  Format)  ‘Price  Reduced* 

9 

$50 

$60 

338D 

MIL-HDBK-338B  (Draft)  (Macintosh  Format)  ‘Price  Reduced* 

9 

$50 

NPRD-P 

NPRD-95  PC  Version  (with  Informix  run-time  module  and  hard  copy  of 
book  also  Included) 

9 

$475 

$515 

NPRD-P 

NPRD-95  PC  Version  (without  run-time  module  for  those  owning 
NPRD-95  software  -  hard  copv  of  book  also  Included) 

9 

$275 

$315 

NRPS 

Nonoperatina  Reliabilitv  Prediction  Svstem  ‘Price  Reduced* 

9 

$350 

VPRED 

VHSIC  Reliabilitv  Prediction  Software 

9 

$100 

$120 

Concurrent  Engineering  Series 


ITCE 

Introduction  to  Concurrent  Enaineerina 

10 

$85 

WCCA 

Worst  Case  Circuit  Analysis  Application  Guidelines 

10 

$85 

FMECA 

Failure  Mode.  Effects  and  Criticalitv  Anaivsis 

10 

B^9 

FTA 

Fault  Tree  Anaivsis  Application  Guide 

10 

$75 

$85 

Shipping  and  Handling  (Note;  Different  postal  rates  for  NPRD  and  VZAP)  -  See  Back 

Quantity  Discount  -  See  Back 

□  ZRG  RAC  Journal  (Free)  Order  T otal 

RCode;  BENCH 


Ordering  Information 


Ordering 

Fax  to  (315)  337-9932  or  mail  to  Reliability  Analysis  Center,  P.0,  Box  4700,  Rome,  NY,  13442-4700. 

Prepayment  is  preferred.  Credit  cards  (VISA,  AMEX,  MasterCard)  are  accepted  for  purchases  of  $25  and  up.  All  Non-US 
orders  must  be  accompanied  by  a  check  drawn  on  a  US  bank.  Make  checks  payable  to  IITRI/RAC. 


Shipping  &  Handling 

US  orders  add  $4.00  per  book,  $6.00  for  First  Class.  Non-US  add  $10.00  per  book  (allow  8-10  weeks  delivery)  for 
surface  mail,  $15.00  per  book  for  air  mail  ($25.00  for  NPRD  or  VZAP). 

Quantity  Discounts 

Discounts  are  available  for  10-f-  copies.  For  discount  prices  call  (800)  526-4802  or  (315)  339-7047. 

Military  Agencies 

Blanket  Purchase  Agreement,  DD  Form  1155,  may  be  used  for  ordering  RAC  products  and  services.  Indicate  the 
maximum  amount  authorized  and  cutoff  date  and  specify  products  and  services  to  be  provided.  Identify  vendor  as 
IIT  Research  institute/Reliability  Analysis  Center. 


To  place  an  order 

Write  to:  Reliability  Analysis  Center,  P.O.  Box  4700,  Rome,  NY  13442-4700 
Call:  (800)  526-4802,  (315)  339-7047 

Fax:  (315)  337-9932 


E-mail:  rac@mail.iitri.com 


Name  _ 

Company  _ 

Division  _ 

Address _ _ 

City  _ State _ Zip 

Country  Phone  Ext. 

Method  of  Payment  _ -  -  - 

□  Personal  check  enclosed 

□  Company  check  enclosed  (Make  checks  payable  to  IITRI/RAC) 


□  Credit  Card  # 

Expiration  Date 

Type  (circle): 

AMERICAN  EXPRESS 

VISA 

MASTERCARD 

Name  on  card: 

Billing  Address: 

□  DD1 155  (Government  personnel) 

□  Company  Purchase  Order 

□  Place  my  name  on  the  distribution  list  for  the  free  RAC  Journal 


RCode:  BENCH 


